diff mbox series

backup: Fix broken globbing expansion

Message ID	20220329122711.558230-1-michael.tremer@ipfire.org
State	Accepted
Commit	13f6473a4f2fc2678a8d1199fc6dcb542c95fff8
Headers	From: Michael Tremer <michael.tremer@ipfire.org> To: development@lists.ipfire.org Subject: [PATCH] backup: Fix broken globbing expansion Date: Tue, 29 Mar 2022 12:27:11 +0000 Message-Id: <20220329122711.558230-1-michael.tremer@ipfire.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Precedence: list Cc: Michael Tremer <michael.tremer@ipfire.org> Errors-To: development-bounces@lists.ipfire.org Sender: "Development" <development-bounces@lists.ipfire.org>
Series	backup: Fix broken globbing expansion \| backup: Fix broken globbing expansion

Commit Message

Michael Tremer March 29, 2022, 12:27 p.m. UTC

  This patch fixes globbing expansion in the backup include file list
which got broken in c7e0d73e7cfd7be95db9d0a5f3392b8241813d5b.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
---
 config/backup/backup.pl | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

Comments

Adolf Belka March 29, 2022, 1:10 p.m. UTC | #1

Hi Michael,

Just tried this patch out on my vm testbed system and it still doesn't work for me. The backup file had got to 1.3GB when I deleted the backup file as it was still growing. The normal correct backup file on that vm machine is around 7MB

The above was the case for both running it from the WUI or from the command line from my unprivileged user using sudo backupctrl exclude

Regards,

Adolf.


On 29/03/2022 14:27, Michael Tremer wrote:
> This patch fixes globbing expansion in the backup include file list
> which got broken in c7e0d73e7cfd7be95db9d0a5f3392b8241813d5b.
>
> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
> ---
>   config/backup/backup.pl | 8 ++++----
>   1 file changed, 4 insertions(+), 4 deletions(-)
>
> diff --git a/config/backup/backup.pl b/config/backup/backup.pl
> index a2337cf23..6f9295e94 100644
> --- a/config/backup/backup.pl
> +++ b/config/backup/backup.pl
> @@ -19,6 +19,8 @@
>   #                                                                             #
>   ###############################################################################
>   
> +shopt -s nullglob
> +
>   NOW="$(date "+%Y-%m-%d-%H:%M")"
>   
>   list_addons() {
> @@ -38,10 +40,8 @@ process_includes() {
>   	for include in $@; do
>   		local file
>   		while read -r file; do
> -			for file in ${file}; do
> -				if [ -e "/${file}" ]; then
> -					echo "${file}"
> -				fi
> +			for file in /${file}; do
> +				echo "${file}"
>   			done
>   		done < "${include}"
>   	done | sort -u

Michael Tremer March 29, 2022, 1:11 p.m. UTC | #2

Hello,

Could you please send the file listing to find out what is being included what shouldn’t?

-Michael

> On 29 Mar 2022, at 14:10, Adolf Belka <adolf.belka@ipfire.org> wrote:
> 
> Hi Michael,
> 
> Just tried this patch out on my vm testbed system and it still doesn't work for me. The backup file had got to 1.3GB when I deleted the backup file as it was still growing. The normal correct backup file on that vm machine is around 7MB
> 
> The above was the case for both running it from the WUI or from the command line from my unprivileged user using sudo backupctrl exclude
> 
> Regards,
> 
> Adolf.
> 
> 
> On 29/03/2022 14:27, Michael Tremer wrote:
>> This patch fixes globbing expansion in the backup include file list
>> which got broken in c7e0d73e7cfd7be95db9d0a5f3392b8241813d5b.
>> 
>> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
>> ---
>>  config/backup/backup.pl | 8 ++++----
>>  1 file changed, 4 insertions(+), 4 deletions(-)
>> 
>> diff --git a/config/backup/backup.pl b/config/backup/backup.pl
>> index a2337cf23..6f9295e94 100644
>> --- a/config/backup/backup.pl
>> +++ b/config/backup/backup.pl
>> @@ -19,6 +19,8 @@
>>  #                                                                             #
>>  ###############################################################################
>>  +shopt -s nullglob
>> +
>>  NOW="$(date "+%Y-%m-%d-%H:%M")"
>>    list_addons() {
>> @@ -38,10 +40,8 @@ process_includes() {
>>  	for include in $@; do
>>  		local file
>>  		while read -r file; do
>> -			for file in ${file}; do
>> -				if [ -e "/${file}" ]; then
>> -					echo "${file}"
>> -				fi
>> +			for file in /${file}; do
>> +				echo "${file}"
>>  			done
>>  		done < "${include}"
>>  	done | sort -u

Bernhard Bitsch March 29, 2022, 1:36 p.m. UTC | #3

Hi,

Am 29.03.2022 um 15:11 schrieb Michael Tremer:
> Hello,
> 
> Could you please send the file listing to find out what is being included what shouldn’t?
> 

Just for easy quick handling, you can use
'/var/ipfire/backup/bin/backup.pl list' to show the files included.

Bernhard
> -Michael
> 
>> On 29 Mar 2022, at 14:10, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>
>> Hi Michael,
>>
>> Just tried this patch out on my vm testbed system and it still doesn't work for me. The backup file had got to 1.3GB when I deleted the backup file as it was still growing. The normal correct backup file on that vm machine is around 7MB
>>
>> The above was the case for both running it from the WUI or from the command line from my unprivileged user using sudo backupctrl exclude
>>
>> Regards,
>>
>> Adolf.
>>
>>
>> On 29/03/2022 14:27, Michael Tremer wrote:
>>> This patch fixes globbing expansion in the backup include file list
>>> which got broken in c7e0d73e7cfd7be95db9d0a5f3392b8241813d5b.
>>>
>>> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
>>> ---
>>>   config/backup/backup.pl | 8 ++++----
>>>   1 file changed, 4 insertions(+), 4 deletions(-)
>>>
>>> diff --git a/config/backup/backup.pl b/config/backup/backup.pl
>>> index a2337cf23..6f9295e94 100644
>>> --- a/config/backup/backup.pl
>>> +++ b/config/backup/backup.pl
>>> @@ -19,6 +19,8 @@
>>>   #                                                                             #
>>>   ###############################################################################
>>>   +shopt -s nullglob
>>> +
>>>   NOW="$(date "+%Y-%m-%d-%H:%M")"
>>>     list_addons() {
>>> @@ -38,10 +40,8 @@ process_includes() {
>>>   	for include in $@; do
>>>   		local file
>>>   		while read -r file; do
>>> -			for file in ${file}; do
>>> -				if [ -e "/${file}" ]; then
>>> -					echo "${file}"
>>> -				fi
>>> +			for file in /${file}; do
>>> +				echo "${file}"
>>>   			done
>>>   		done < "${include}"
>>>   	done | sort -u
>

Michael Tremer March 29, 2022, 1:49 p.m. UTC | #4

That is what I have been using to check if this works fine.

> On 29 Mar 2022, at 14:36, Bernhard Bitsch <bbitsch@ipfire.org> wrote:
> 
> Hi,
> 
> Am 29.03.2022 um 15:11 schrieb Michael Tremer:
>> Hello,
>> Could you please send the file listing to find out what is being included what shouldn’t?
> 
> Just for easy quick handling, you can use
> '/var/ipfire/backup/bin/backup.pl list' to show the files included.
> 
> Bernhard
>> -Michael
>>> On 29 Mar 2022, at 14:10, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>> 
>>> Hi Michael,
>>> 
>>> Just tried this patch out on my vm testbed system and it still doesn't work for me. The backup file had got to 1.3GB when I deleted the backup file as it was still growing. The normal correct backup file on that vm machine is around 7MB
>>> 
>>> The above was the case for both running it from the WUI or from the command line from my unprivileged user using sudo backupctrl exclude
>>> 
>>> Regards,
>>> 
>>> Adolf.
>>> 
>>> 
>>> On 29/03/2022 14:27, Michael Tremer wrote:
>>>> This patch fixes globbing expansion in the backup include file list
>>>> which got broken in c7e0d73e7cfd7be95db9d0a5f3392b8241813d5b.
>>>> 
>>>> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
>>>> ---
>>>>  config/backup/backup.pl | 8 ++++----
>>>>  1 file changed, 4 insertions(+), 4 deletions(-)
>>>> 
>>>> diff --git a/config/backup/backup.pl b/config/backup/backup.pl
>>>> index a2337cf23..6f9295e94 100644
>>>> --- a/config/backup/backup.pl
>>>> +++ b/config/backup/backup.pl
>>>> @@ -19,6 +19,8 @@
>>>>  #                                                                             #
>>>>  ###############################################################################
>>>>  +shopt -s nullglob
>>>> +
>>>>  NOW="$(date "+%Y-%m-%d-%H:%M")"
>>>>    list_addons() {
>>>> @@ -38,10 +40,8 @@ process_includes() {
>>>>  	for include in $@; do
>>>>  		local file
>>>>  		while read -r file; do
>>>> -			for file in ${file}; do
>>>> -				if [ -e "/${file}" ]; then
>>>> -					echo "${file}"
>>>> -				fi
>>>> +			for file in /${file}; do
>>>> +				echo "${file}"
>>>>  			done
>>>>  		done < "${include}"
>>>>  	done | sort -u

Adolf Belka March 29, 2022, 2:01 p.m. UTC | #5

Hi All,

On 29/03/2022 15:36, Bernhard Bitsch wrote:
> Hi,
> 
> Am 29.03.2022 um 15:11 schrieb Michael Tremer:
>> Hello,
>>
>> Could you please send the file listing to find out what is being included what shouldn’t?
Unfortunately, as I stopped the backup continuing once it had reached 1.2GB, the file created was not able to be opened. Probably stopping the backup corrupted it in some way.
>>
> 
> Just for easy quick handling, you can use
> '/var/ipfire/backup/bin/backup.pl list' to show the files included.
I gave this a go and here is the output from the command.

-bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
/
/etc/conntrackd/conntrackd.conf
/etc/group
/etc/hosts
/etc/hosts.allow
/etc/hosts.deny
/etc/httpd/server.crt
/etc/httpd/server.csr
/etc/httpd/server-ecdsa.crt
/etc/httpd/server-ecdsa.csr
/etc/httpd/server-ecdsa.key
/etc/httpd/server.key
/etc/ipsec.user.conf
/etc/ipsec.user-post.conf
/etc/ipsec.user.secrets
/etc/logrotate.d
/etc/passwd
/etc/shadow
/etc/squid/squid.conf.local
/etc/squid/squid.conf.pre.local
/etc/ssh/sshd_config
/etc/ssh/ssh_host_ecdsa_key
/etc/ssh/ssh_host_ecdsa_key.pub
/etc/ssh/ssh_host_ed25519_key
/etc/ssh/ssh_host_ed25519_key.pub
/etc/ssh/ssh_host_rsa_key
/etc/ssh/ssh_host_rsa_key.pub
//etc/sudoers
/etc/sysconfig/createfiles
/etc/sysconfig/firewall.local
/etc/sysconfig/lm_sensors
/etc/sysconfig/modules
/etc/sysconfig/ramdisk
/etc/sysconfig/rc
/etc/sysconfig/rc.local
/etc/unbound
//home/ahb
/root/.bash_history
/root/.gitconfig
/root/.ssh
/var/ipfire/accounting/settings.conf
/var/ipfire/auth/users
/var/ipfire/backup/addons/backup
/var/ipfire/backup/exclude.user
/var/ipfire/backup/include.user
/var/ipfire/ca/cacert.pem
/var/ipfire/captive/agb.txt
/var/ipfire/captive/clients
/var/ipfire/captive/coupons
/var/ipfire/captive/logo.dat
/var/ipfire/captive/settings
/var/ipfire/captive/terms.txt
/var/ipfire/captive/voucher_out
/var/ipfire/certs/hostcert.pem
/var/ipfire/certs/hostkey.pem
/var/ipfire/certs/phoebevmipseccert.pem
/var/ipfire/connscheduler/connscheduler.conf
/var/ipfire/crls/cacrl.pem
/var/ipfire/cups/cups-browsed.conf
/var/ipfire/cups/subscriptions.conf
/var/ipfire/ddns/config
/var/ipfire/ddns/ddns.conf
/var/ipfire/ddns/settings
/var/ipfire/dhcp/advoptions
/var/ipfire/dhcp/advoptions-list
/var/ipfire/dhcpc/dhcpcd.conf
/var/ipfire/dhcp/dhcpd.conf
/var/ipfire/dhcp/dhcpd.conf.local
/var/ipfire/dhcp/enable_blue
/var/ipfire/dhcp/enable_green
/var/ipfire/dhcp/fixleases
/var/ipfire/dhcp/settings
/var/ipfire/dma/auth.conf
/var/ipfire/dma/dma.conf
/var/ipfire/dma/mail.conf
/var/ipfire/dns
/var/ipfire/dnsforward/config
/var/ipfire/dns/settings
/var/ipfire/ethernet/aliases
/var/ipfire/ethernet/settings
/var/ipfire/ethernet/wireless
/var/ipfire/extrahd/settings
/var/ipfire/firewall
/var/ipfire/firewall/config
/var/ipfire/firewall/settings
/var/ipfire/fwhosts
/var/ipfire/isdn/settings
/var/ipfire/logging/settings
/var/ipfire/mac/settings
/var/ipfire/main/firstsetup_ok
/var/ipfire/main/gpl_accepted
/var/ipfire/main/hostname.conf
/var/ipfire/main/hosts
/var/ipfire/main/manualpages
/var/ipfire/main/routing
/var/ipfire/main/security
/var/ipfire/main/send_profile
/var/ipfire/main/settings
/var/ipfire/modem/settings
/var/ipfire/optionsfw/settings
/var/ipfire/ovpn
/var/ipfire/ovpn/ccd.conf
/var/ipfire/ovpn/collectd.vpn
/var/ipfire/ovpn/enable
/var/ipfire/ovpn/server.conf
/var/ipfire/ovpn/settings
/var/ipfire/pakfire/settings
/var/ipfire/ppp
/var/ipfire/ppp/fake-resolv.conf
/var/ipfire/ppp/settings
/var/ipfire/private/cakey.pem
/var/ipfire/proxy
/var/ipfire/proxy/asnbl-helper.conf
/var/ipfire/proxy/cachemgr.conf
/var/ipfire/proxy/enable
/var/ipfire/proxy/settings
/var/ipfire/proxy/squid.conf
/var/ipfire/qos/bin
/var/ipfire/qos/bin/qos.sh
/var/ipfire/qos/classes
/var/ipfire/qos/level7config
/var/ipfire/qos/portconfig
/var/ipfire/qos/settings
/var/ipfire/qos/subclasses
/var/ipfire/qos/tosconfig
/var/ipfire/remote/enablessh
/var/ipfire/remote/settings
/var/ipfire/sensors/settings
/var/ipfire/suricata/oinkmaster.conf
/var/ipfire/suricata/oinkmaster-modify-sids.conf
/var/ipfire/suricata/oinkmaster-provider-includes.conf
/var/ipfire/suricata/providers-settings
/var/ipfire/suricata/settings
/var/ipfire/suricata/suricata-default-rules.yaml
/var/ipfire/suricata/suricata-dns-servers.yaml
/var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
/var/ipfire/suricata/suricata-homenet.yaml
/var/ipfire/suricata/suricata-http-ports.yaml
/var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml
/var/ipfire/suricata/suricata-used-providers.yaml
/var/ipfire/time/
/var/ipfire/time/counter.conf
/var/ipfire/time/enable
/var/ipfire/time/settime.conf
/var/ipfire/time/settings
/var/ipfire/upnp/settings
/var/ipfire/urlfilter
/var/ipfire/urlfilter/settings
/var/ipfire/urlfilter/squidGuard.conf
/var/ipfire/vpn
/var/ipfire/vpn/config
/var/ipfire/vpn/ipsec.conf
/var/ipfire/vpn/settings
/var/ipfire/wakeonlan/clients.conf
/var/ipfire/wio/wio.conf
/var/ipfire/wireless/config
/var/ipfire/wireless/settings
/var/lib/suricata
/var/log/rrd/collectd
/var/log/rrd/hddshutdown-md127.rrd
/var/log/rrd/hddshutdown-sda.rrd
/var/log/rrd/hddshutdown-sdb.rrd
/var/log/rrd/hddtemp-md127.rrd
/var/log/rrd/hddtemp-sda.rrd
/var/log/rrd/hddtemp-sdb.rrd
/var/log/rrd/wio
/var/log/vnstat
/var/tmp/idsrules-emerging.tar.gz
/var/tmp/idsrules-sslbl_blacklist.rules

Regards,
Adolf.
> 
> Bernhard
>> -Michael
>>
>>> On 29 Mar 2022, at 14:10, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>>
>>> Hi Michael,
>>>
>>> Just tried this patch out on my vm testbed system and it still doesn't work for me. The backup file had got to 1.3GB when I deleted the backup file as it was still growing. The normal correct backup file on that vm machine is around 7MB
>>>
>>> The above was the case for both running it from the WUI or from the command line from my unprivileged user using sudo backupctrl exclude
>>>
>>> Regards,
>>>
>>> Adolf.
>>>
>>>
>>> On 29/03/2022 14:27, Michael Tremer wrote:
>>>> This patch fixes globbing expansion in the backup include file list
>>>> which got broken in c7e0d73e7cfd7be95db9d0a5f3392b8241813d5b.
>>>>
>>>> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
>>>> ---
>>>>   config/backup/backup.pl | 8 ++++----
>>>>   1 file changed, 4 insertions(+), 4 deletions(-)
>>>>
>>>> diff --git a/config/backup/backup.pl b/config/backup/backup.pl
>>>> index a2337cf23..6f9295e94 100644
>>>> --- a/config/backup/backup.pl
>>>> +++ b/config/backup/backup.pl
>>>> @@ -19,6 +19,8 @@
>>>>   #                                                                             #
>>>>   ###############################################################################
>>>>   +shopt -s nullglob
>>>> +
>>>>   NOW="$(date "+%Y-%m-%d-%H:%M")"
>>>>     list_addons() {
>>>> @@ -38,10 +40,8 @@ process_includes() {
>>>>       for include in $@; do
>>>>           local file
>>>>           while read -r file; do
>>>> -            for file in ${file}; do
>>>> -                if [ -e "/${file}" ]; then
>>>> -                    echo "${file}"
>>>> -                fi
>>>> +            for file in /${file}; do
>>>> +                echo "${file}"
>>>>               done
>>>>           done < "${include}"
>>>>       done | sort -u
>>

Michael Tremer March 29, 2022, 2:03 p.m. UTC | #6

Hey,

> On 29 Mar 2022, at 15:01, Adolf Belka <adolf.belka@ipfire.org> wrote:
> 
> Hi All,
> 
> On 29/03/2022 15:36, Bernhard Bitsch wrote:
>> Hi,
>> Am 29.03.2022 um 15:11 schrieb Michael Tremer:
>>> Hello,
>>> 
>>> Could you please send the file listing to find out what is being included what shouldn’t?
> Unfortunately, as I stopped the backup continuing once it had reached 1.2GB, the file created was not able to be opened. Probably stopping the backup corrupted it in some way.
>>> 
>> Just for easy quick handling, you can use
>> '/var/ipfire/backup/bin/backup.pl list' to show the files included.
> I gave this a go and here is the output from the command.
> 
> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
> /

You have / on the list which will cause that the entire system is being put into the backup.

Do you have anything custom in include.user?

> /etc/conntrackd/conntrackd.conf
> /etc/group
> /etc/hosts
> /etc/hosts.allow
> /etc/hosts.deny
> /etc/httpd/server.crt
> /etc/httpd/server.csr
> /etc/httpd/server-ecdsa.crt
> /etc/httpd/server-ecdsa.csr
> /etc/httpd/server-ecdsa.key
> /etc/httpd/server.key
> /etc/ipsec.user.conf
> /etc/ipsec.user-post.conf
> /etc/ipsec.user.secrets
> /etc/logrotate.d
> /etc/passwd
> /etc/shadow
> /etc/squid/squid.conf.local
> /etc/squid/squid.conf.pre.local
> /etc/ssh/sshd_config
> /etc/ssh/ssh_host_ecdsa_key
> /etc/ssh/ssh_host_ecdsa_key.pub
> /etc/ssh/ssh_host_ed25519_key
> /etc/ssh/ssh_host_ed25519_key.pub
> /etc/ssh/ssh_host_rsa_key
> /etc/ssh/ssh_host_rsa_key.pub
> //etc/sudoers
> /etc/sysconfig/createfiles
> /etc/sysconfig/firewall.local
> /etc/sysconfig/lm_sensors
> /etc/sysconfig/modules
> /etc/sysconfig/ramdisk
> /etc/sysconfig/rc
> /etc/sysconfig/rc.local
> /etc/unbound
> //home/ahb
> /root/.bash_history
> /root/.gitconfig
> /root/.ssh
> /var/ipfire/accounting/settings.conf
> /var/ipfire/auth/users
> /var/ipfire/backup/addons/backup
> /var/ipfire/backup/exclude.user
> /var/ipfire/backup/include.user
> /var/ipfire/ca/cacert.pem
> /var/ipfire/captive/agb.txt
> /var/ipfire/captive/clients
> /var/ipfire/captive/coupons
> /var/ipfire/captive/logo.dat
> /var/ipfire/captive/settings
> /var/ipfire/captive/terms.txt
> /var/ipfire/captive/voucher_out
> /var/ipfire/certs/hostcert.pem
> /var/ipfire/certs/hostkey.pem
> /var/ipfire/certs/phoebevmipseccert.pem
> /var/ipfire/connscheduler/connscheduler.conf
> /var/ipfire/crls/cacrl.pem
> /var/ipfire/cups/cups-browsed.conf
> /var/ipfire/cups/subscriptions.conf
> /var/ipfire/ddns/config
> /var/ipfire/ddns/ddns.conf
> /var/ipfire/ddns/settings
> /var/ipfire/dhcp/advoptions
> /var/ipfire/dhcp/advoptions-list
> /var/ipfire/dhcpc/dhcpcd.conf
> /var/ipfire/dhcp/dhcpd.conf
> /var/ipfire/dhcp/dhcpd.conf.local
> /var/ipfire/dhcp/enable_blue
> /var/ipfire/dhcp/enable_green
> /var/ipfire/dhcp/fixleases
> /var/ipfire/dhcp/settings
> /var/ipfire/dma/auth.conf
> /var/ipfire/dma/dma.conf
> /var/ipfire/dma/mail.conf
> /var/ipfire/dns
> /var/ipfire/dnsforward/config
> /var/ipfire/dns/settings
> /var/ipfire/ethernet/aliases
> /var/ipfire/ethernet/settings
> /var/ipfire/ethernet/wireless
> /var/ipfire/extrahd/settings
> /var/ipfire/firewall
> /var/ipfire/firewall/config
> /var/ipfire/firewall/settings
> /var/ipfire/fwhosts
> /var/ipfire/isdn/settings
> /var/ipfire/logging/settings
> /var/ipfire/mac/settings
> /var/ipfire/main/firstsetup_ok
> /var/ipfire/main/gpl_accepted
> /var/ipfire/main/hostname.conf
> /var/ipfire/main/hosts
> /var/ipfire/main/manualpages
> /var/ipfire/main/routing
> /var/ipfire/main/security
> /var/ipfire/main/send_profile
> /var/ipfire/main/settings
> /var/ipfire/modem/settings
> /var/ipfire/optionsfw/settings
> /var/ipfire/ovpn
> /var/ipfire/ovpn/ccd.conf
> /var/ipfire/ovpn/collectd.vpn
> /var/ipfire/ovpn/enable
> /var/ipfire/ovpn/server.conf
> /var/ipfire/ovpn/settings
> /var/ipfire/pakfire/settings
> /var/ipfire/ppp
> /var/ipfire/ppp/fake-resolv.conf
> /var/ipfire/ppp/settings
> /var/ipfire/private/cakey.pem
> /var/ipfire/proxy
> /var/ipfire/proxy/asnbl-helper.conf
> /var/ipfire/proxy/cachemgr.conf
> /var/ipfire/proxy/enable
> /var/ipfire/proxy/settings
> /var/ipfire/proxy/squid.conf
> /var/ipfire/qos/bin
> /var/ipfire/qos/bin/qos.sh
> /var/ipfire/qos/classes
> /var/ipfire/qos/level7config
> /var/ipfire/qos/portconfig
> /var/ipfire/qos/settings
> /var/ipfire/qos/subclasses
> /var/ipfire/qos/tosconfig
> /var/ipfire/remote/enablessh
> /var/ipfire/remote/settings
> /var/ipfire/sensors/settings
> /var/ipfire/suricata/oinkmaster.conf
> /var/ipfire/suricata/oinkmaster-modify-sids.conf
> /var/ipfire/suricata/oinkmaster-provider-includes.conf
> /var/ipfire/suricata/providers-settings
> /var/ipfire/suricata/settings
> /var/ipfire/suricata/suricata-default-rules.yaml
> /var/ipfire/suricata/suricata-dns-servers.yaml
> /var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
> /var/ipfire/suricata/suricata-homenet.yaml
> /var/ipfire/suricata/suricata-http-ports.yaml
> /var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml
> /var/ipfire/suricata/suricata-used-providers.yaml
> /var/ipfire/time/
> /var/ipfire/time/counter.conf
> /var/ipfire/time/enable
> /var/ipfire/time/settime.conf
> /var/ipfire/time/settings
> /var/ipfire/upnp/settings
> /var/ipfire/urlfilter
> /var/ipfire/urlfilter/settings
> /var/ipfire/urlfilter/squidGuard.conf
> /var/ipfire/vpn
> /var/ipfire/vpn/config
> /var/ipfire/vpn/ipsec.conf
> /var/ipfire/vpn/settings
> /var/ipfire/wakeonlan/clients.conf
> /var/ipfire/wio/wio.conf
> /var/ipfire/wireless/config
> /var/ipfire/wireless/settings
> /var/lib/suricata
> /var/log/rrd/collectd
> /var/log/rrd/hddshutdown-md127.rrd
> /var/log/rrd/hddshutdown-sda.rrd
> /var/log/rrd/hddshutdown-sdb.rrd
> /var/log/rrd/hddtemp-md127.rrd
> /var/log/rrd/hddtemp-sda.rrd
> /var/log/rrd/hddtemp-sdb.rrd
> /var/log/rrd/wio
> /var/log/vnstat
> /var/tmp/idsrules-emerging.tar.gz
> /var/tmp/idsrules-sslbl_blacklist.rules
> 
> Regards,
> Adolf.
>> Bernhard
>>> -Michael
>>> 
>>>> On 29 Mar 2022, at 14:10, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>>> 
>>>> Hi Michael,
>>>> 
>>>> Just tried this patch out on my vm testbed system and it still doesn't work for me. The backup file had got to 1.3GB when I deleted the backup file as it was still growing. The normal correct backup file on that vm machine is around 7MB
>>>> 
>>>> The above was the case for both running it from the WUI or from the command line from my unprivileged user using sudo backupctrl exclude
>>>> 
>>>> Regards,
>>>> 
>>>> Adolf.
>>>> 
>>>> 
>>>> On 29/03/2022 14:27, Michael Tremer wrote:
>>>>> This patch fixes globbing expansion in the backup include file list
>>>>> which got broken in c7e0d73e7cfd7be95db9d0a5f3392b8241813d5b.
>>>>> 
>>>>> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
>>>>> ---
>>>>>   config/backup/backup.pl | 8 ++++----
>>>>>   1 file changed, 4 insertions(+), 4 deletions(-)
>>>>> 
>>>>> diff --git a/config/backup/backup.pl b/config/backup/backup.pl
>>>>> index a2337cf23..6f9295e94 100644
>>>>> --- a/config/backup/backup.pl
>>>>> +++ b/config/backup/backup.pl
>>>>> @@ -19,6 +19,8 @@
>>>>>   #                                                                             #
>>>>>   ###############################################################################
>>>>>   +shopt -s nullglob
>>>>> +
>>>>>   NOW="$(date "+%Y-%m-%d-%H:%M")"
>>>>>     list_addons() {
>>>>> @@ -38,10 +40,8 @@ process_includes() {
>>>>>       for include in $@; do
>>>>>           local file
>>>>>           while read -r file; do
>>>>> -            for file in ${file}; do
>>>>> -                if [ -e "/${file}" ]; then
>>>>> -                    echo "${file}"
>>>>> -                fi
>>>>> +            for file in /${file}; do
>>>>> +                echo "${file}"
>>>>>               done
>>>>>           done < "${include}"
>>>>>       done | sort -u
>>>

Adolf Belka March 29, 2022, 2:14 p.m. UTC | #7

Hi All,

This is what I get with the pushd/popd version in place on the same system

-bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
etc/group
etc/hosts
etc/hosts.allow
etc/hosts.deny
etc/httpd/server.crt
etc/httpd/server.csr
etc/httpd/server-ecdsa.crt
etc/httpd/server-ecdsa.csr
etc/httpd/server-ecdsa.key
etc/httpd/server.key
etc/ipsec.user.conf
etc/ipsec.user.secrets
etc/logrotate.d
etc/passwd
etc/shadow
etc/ssh/sshd_config
etc/ssh/ssh_host_ecdsa_key
etc/ssh/ssh_host_ecdsa_key.pub
etc/ssh/ssh_host_ed25519_key
etc/ssh/ssh_host_ed25519_key.pub
etc/ssh/ssh_host_rsa_key
etc/ssh/ssh_host_rsa_key.pub
/etc/sudoers
etc/sysconfig/createfiles
etc/sysconfig/firewall.local
etc/sysconfig/lm_sensors
etc/sysconfig/modules
etc/sysconfig/ramdisk
etc/sysconfig/rc
etc/sysconfig/rc.local
etc/unbound
/home/ahb
root/.bash_history
var/ipfire/accounting/settings.conf
var/ipfire/auth/users
var/ipfire/backup/addons/backup
var/ipfire/backup/exclude.user
var/ipfire/backup/include.user
var/ipfire/ca/cacert.pem
var/ipfire/captive/agb.txt
var/ipfire/captive/clients
var/ipfire/captive/coupons
var/ipfire/captive/logo.dat
var/ipfire/captive/settings
var/ipfire/captive/terms.txt
var/ipfire/captive/voucher_out
var/ipfire/certs/hostcert.pem
var/ipfire/certs/hostkey.pem
var/ipfire/certs/phoebevmipseccert.pem
var/ipfire/connscheduler/connscheduler.conf
var/ipfire/crls/cacrl.pem
var/ipfire/cups/cups-browsed.conf
var/ipfire/cups/subscriptions.conf
var/ipfire/ddns/config
var/ipfire/ddns/ddns.conf
var/ipfire/ddns/settings
var/ipfire/dhcp/advoptions
var/ipfire/dhcp/advoptions-list
var/ipfire/dhcpc/dhcpcd.conf
var/ipfire/dhcp/dhcpd.conf
var/ipfire/dhcp/dhcpd.conf.local
var/ipfire/dhcp/enable_blue
var/ipfire/dhcp/enable_green
var/ipfire/dhcp/fixleases
var/ipfire/dhcp/settings
var/ipfire/dma/auth.conf
var/ipfire/dma/dma.conf
var/ipfire/dma/mail.conf
var/ipfire/dns
var/ipfire/dnsforward/config
var/ipfire/dns/settings
var/ipfire/ethernet/aliases
var/ipfire/ethernet/settings
var/ipfire/ethernet/wireless
var/ipfire/extrahd/settings
var/ipfire/firewall
var/ipfire/firewall/config
var/ipfire/firewall/settings
var/ipfire/fwhosts
var/ipfire/isdn/settings
var/ipfire/logging/settings
var/ipfire/mac/settings
var/ipfire/main/firstsetup_ok
var/ipfire/main/gpl_accepted
var/ipfire/main/hostname.conf
var/ipfire/main/hosts
var/ipfire/main/manualpages
var/ipfire/main/routing
var/ipfire/main/security
var/ipfire/main/send_profile
var/ipfire/main/settings
var/ipfire/modem/settings
var/ipfire/optionsfw/settings
var/ipfire/ovpn
var/ipfire/ovpn/ccd.conf
var/ipfire/ovpn/collectd.vpn
var/ipfire/ovpn/enable
var/ipfire/ovpn/server.conf
var/ipfire/ovpn/settings
var/ipfire/pakfire/settings
var/ipfire/ppp
var/ipfire/ppp/fake-resolv.conf
var/ipfire/ppp/settings
var/ipfire/private/cakey.pem
var/ipfire/proxy
var/ipfire/proxy/asnbl-helper.conf
var/ipfire/proxy/cachemgr.conf
var/ipfire/proxy/enable
var/ipfire/proxy/settings
var/ipfire/proxy/squid.conf
var/ipfire/qos/bin
var/ipfire/qos/classes
var/ipfire/qos/level7config
var/ipfire/qos/portconfig
var/ipfire/qos/settings
var/ipfire/qos/subclasses
var/ipfire/qos/tosconfig
var/ipfire/remote/enablessh
var/ipfire/remote/settings
var/ipfire/sensors/settings
var/ipfire/suricata/oinkmaster.conf
var/ipfire/suricata/oinkmaster-modify-sids.conf
var/ipfire/suricata/oinkmaster-provider-includes.conf
var/ipfire/suricata/providers-settings
var/ipfire/suricata/settings
var/ipfire/suricata/suricata-default-rules.yaml
var/ipfire/suricata/suricata-dns-servers.yaml
var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
var/ipfire/suricata/suricata-homenet.yaml
var/ipfire/suricata/suricata-http-ports.yaml
var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml
var/ipfire/suricata/suricata-used-providers.yaml
var/ipfire/time/
var/ipfire/time/counter.conf
var/ipfire/time/enable
var/ipfire/time/settime.conf
var/ipfire/time/settings
var/ipfire/upnp/settings
var/ipfire/urlfilter
var/ipfire/urlfilter/settings
var/ipfire/urlfilter/squidGuard.conf
var/ipfire/vpn
var/ipfire/vpn/config
var/ipfire/vpn/ipsec.conf
var/ipfire/vpn/settings
var/ipfire/wakeonlan/clients.conf
var/ipfire/wio/wio.conf
var/ipfire/wireless/config
var/ipfire/wireless/settings
var/lib/suricata
var/log/rrd/collectd
var/log/rrd/hddshutdown-md127.rrd
var/log/rrd/hddshutdown-sda.rrd
var/log/rrd/hddshutdown-sdb.rrd
var/log/rrd/hddtemp-md127.rrd
var/log/rrd/hddtemp-sda.rrd
var/log/rrd/hddtemp-sdb.rrd
var/log/rrd/wio
var/log/vnstat
var/tmp/idsrules-emerging.tar.gz
var/tmp/idsrules-sslbl_blacklist.rules

The following are in the previous list but not in this one:-

/
/etc/conntrackd/conntrackd.conf
/etc/ipsec.user-post.conf
/root/.gitconfig
/root/.ssh

Regards,

Adolf.

On 29/03/2022 16:01, Adolf Belka wrote:
> Hi All,
>
> On 29/03/2022 15:36, Bernhard Bitsch wrote:
>> Hi,
>>
>> Am 29.03.2022 um 15:11 schrieb Michael Tremer:
>>> Hello,
>>>
>>> Could you please send the file listing to find out what is being included what shouldn’t?
> Unfortunately, as I stopped the backup continuing once it had reached 1.2GB, the file created was not able to be opened. Probably stopping the backup corrupted it in some way.
>>>
>>
>> Just for easy quick handling, you can use
>> '/var/ipfire/backup/bin/backup.pl list' to show the files included.
> I gave this a go and here is the output from the command.
>
> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
> /
> /etc/conntrackd/conntrackd.conf
> /etc/group
> /etc/hosts
> /etc/hosts.allow
> /etc/hosts.deny
> /etc/httpd/server.crt
> /etc/httpd/server.csr
> /etc/httpd/server-ecdsa.crt
> /etc/httpd/server-ecdsa.csr
> /etc/httpd/server-ecdsa.key
> /etc/httpd/server.key
> /etc/ipsec.user.conf
> /etc/ipsec.user-post.conf
> /etc/ipsec.user.secrets
> /etc/logrotate.d
> /etc/passwd
> /etc/shadow
> /etc/squid/squid.conf.local
> /etc/squid/squid.conf.pre.local
> /etc/ssh/sshd_config
> /etc/ssh/ssh_host_ecdsa_key
> /etc/ssh/ssh_host_ecdsa_key.pub
> /etc/ssh/ssh_host_ed25519_key
> /etc/ssh/ssh_host_ed25519_key.pub
> /etc/ssh/ssh_host_rsa_key
> /etc/ssh/ssh_host_rsa_key.pub
> //etc/sudoers
> /etc/sysconfig/createfiles
> /etc/sysconfig/firewall.local
> /etc/sysconfig/lm_sensors
> /etc/sysconfig/modules
> /etc/sysconfig/ramdisk
> /etc/sysconfig/rc
> /etc/sysconfig/rc.local
> /etc/unbound
> //home/ahb
> /root/.bash_history
> /root/.gitconfig
> /root/.ssh
> /var/ipfire/accounting/settings.conf
> /var/ipfire/auth/users
> /var/ipfire/backup/addons/backup
> /var/ipfire/backup/exclude.user
> /var/ipfire/backup/include.user
> /var/ipfire/ca/cacert.pem
> /var/ipfire/captive/agb.txt
> /var/ipfire/captive/clients
> /var/ipfire/captive/coupons
> /var/ipfire/captive/logo.dat
> /var/ipfire/captive/settings
> /var/ipfire/captive/terms.txt
> /var/ipfire/captive/voucher_out
> /var/ipfire/certs/hostcert.pem
> /var/ipfire/certs/hostkey.pem
> /var/ipfire/certs/phoebevmipseccert.pem
> /var/ipfire/connscheduler/connscheduler.conf
> /var/ipfire/crls/cacrl.pem
> /var/ipfire/cups/cups-browsed.conf
> /var/ipfire/cups/subscriptions.conf
> /var/ipfire/ddns/config
> /var/ipfire/ddns/ddns.conf
> /var/ipfire/ddns/settings
> /var/ipfire/dhcp/advoptions
> /var/ipfire/dhcp/advoptions-list
> /var/ipfire/dhcpc/dhcpcd.conf
> /var/ipfire/dhcp/dhcpd.conf
> /var/ipfire/dhcp/dhcpd.conf.local
> /var/ipfire/dhcp/enable_blue
> /var/ipfire/dhcp/enable_green
> /var/ipfire/dhcp/fixleases
> /var/ipfire/dhcp/settings
> /var/ipfire/dma/auth.conf
> /var/ipfire/dma/dma.conf
> /var/ipfire/dma/mail.conf
> /var/ipfire/dns
> /var/ipfire/dnsforward/config
> /var/ipfire/dns/settings
> /var/ipfire/ethernet/aliases
> /var/ipfire/ethernet/settings
> /var/ipfire/ethernet/wireless
> /var/ipfire/extrahd/settings
> /var/ipfire/firewall
> /var/ipfire/firewall/config
> /var/ipfire/firewall/settings
> /var/ipfire/fwhosts
> /var/ipfire/isdn/settings
> /var/ipfire/logging/settings
> /var/ipfire/mac/settings
> /var/ipfire/main/firstsetup_ok
> /var/ipfire/main/gpl_accepted
> /var/ipfire/main/hostname.conf
> /var/ipfire/main/hosts
> /var/ipfire/main/manualpages
> /var/ipfire/main/routing
> /var/ipfire/main/security
> /var/ipfire/main/send_profile
> /var/ipfire/main/settings
> /var/ipfire/modem/settings
> /var/ipfire/optionsfw/settings
> /var/ipfire/ovpn
> /var/ipfire/ovpn/ccd.conf
> /var/ipfire/ovpn/collectd.vpn
> /var/ipfire/ovpn/enable
> /var/ipfire/ovpn/server.conf
> /var/ipfire/ovpn/settings
> /var/ipfire/pakfire/settings
> /var/ipfire/ppp
> /var/ipfire/ppp/fake-resolv.conf
> /var/ipfire/ppp/settings
> /var/ipfire/private/cakey.pem
> /var/ipfire/proxy
> /var/ipfire/proxy/asnbl-helper.conf
> /var/ipfire/proxy/cachemgr.conf
> /var/ipfire/proxy/enable
> /var/ipfire/proxy/settings
> /var/ipfire/proxy/squid.conf
> /var/ipfire/qos/bin
> /var/ipfire/qos/bin/qos.sh
> /var/ipfire/qos/classes
> /var/ipfire/qos/level7config
> /var/ipfire/qos/portconfig
> /var/ipfire/qos/settings
> /var/ipfire/qos/subclasses
> /var/ipfire/qos/tosconfig
> /var/ipfire/remote/enablessh
> /var/ipfire/remote/settings
> /var/ipfire/sensors/settings
> /var/ipfire/suricata/oinkmaster.conf
> /var/ipfire/suricata/oinkmaster-modify-sids.conf
> /var/ipfire/suricata/oinkmaster-provider-includes.conf
> /var/ipfire/suricata/providers-settings
> /var/ipfire/suricata/settings
> /var/ipfire/suricata/suricata-default-rules.yaml
> /var/ipfire/suricata/suricata-dns-servers.yaml
> /var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
> /var/ipfire/suricata/suricata-homenet.yaml
> /var/ipfire/suricata/suricata-http-ports.yaml
> /var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml
> /var/ipfire/suricata/suricata-used-providers.yaml
> /var/ipfire/time/
> /var/ipfire/time/counter.conf
> /var/ipfire/time/enable
> /var/ipfire/time/settime.conf
> /var/ipfire/time/settings
> /var/ipfire/upnp/settings
> /var/ipfire/urlfilter
> /var/ipfire/urlfilter/settings
> /var/ipfire/urlfilter/squidGuard.conf
> /var/ipfire/vpn
> /var/ipfire/vpn/config
> /var/ipfire/vpn/ipsec.conf
> /var/ipfire/vpn/settings
> /var/ipfire/wakeonlan/clients.conf
> /var/ipfire/wio/wio.conf
> /var/ipfire/wireless/config
> /var/ipfire/wireless/settings
> /var/lib/suricata
> /var/log/rrd/collectd
> /var/log/rrd/hddshutdown-md127.rrd
> /var/log/rrd/hddshutdown-sda.rrd
> /var/log/rrd/hddshutdown-sdb.rrd
> /var/log/rrd/hddtemp-md127.rrd
> /var/log/rrd/hddtemp-sda.rrd
> /var/log/rrd/hddtemp-sdb.rrd
> /var/log/rrd/wio
> /var/log/vnstat
> /var/tmp/idsrules-emerging.tar.gz
> /var/tmp/idsrules-sslbl_blacklist.rules
>
> Regards,
> Adolf.
>>
>> Bernhard
>>> -Michael
>>>
>>>> On 29 Mar 2022, at 14:10, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>>>
>>>> Hi Michael,
>>>>
>>>> Just tried this patch out on my vm testbed system and it still doesn't work for me. The backup file had got to 1.3GB when I deleted the backup file as it was still growing. The normal correct backup file on that vm machine is around 7MB
>>>>
>>>> The above was the case for both running it from the WUI or from the command line from my unprivileged user using sudo backupctrl exclude
>>>>
>>>> Regards,
>>>>
>>>> Adolf.
>>>>
>>>>
>>>> On 29/03/2022 14:27, Michael Tremer wrote:
>>>>> This patch fixes globbing expansion in the backup include file list
>>>>> which got broken in c7e0d73e7cfd7be95db9d0a5f3392b8241813d5b.
>>>>>
>>>>> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
>>>>> ---
>>>>>   config/backup/backup.pl | 8 ++++----
>>>>>   1 file changed, 4 insertions(+), 4 deletions(-)
>>>>>
>>>>> diff --git a/config/backup/backup.pl b/config/backup/backup.pl
>>>>> index a2337cf23..6f9295e94 100644
>>>>> --- a/config/backup/backup.pl
>>>>> +++ b/config/backup/backup.pl
>>>>> @@ -19,6 +19,8 @@
>>>>> # #
>>>>> ###############################################################################
>>>>>   +shopt -s nullglob
>>>>> +
>>>>>   NOW="$(date "+%Y-%m-%d-%H:%M")"
>>>>>     list_addons() {
>>>>> @@ -38,10 +40,8 @@ process_includes() {
>>>>>       for include in $@; do
>>>>>           local file
>>>>>           while read -r file; do
>>>>> -            for file in ${file}; do
>>>>> -                if [ -e "/${file}" ]; then
>>>>> -                    echo "${file}"
>>>>> -                fi
>>>>> +            for file in /${file}; do
>>>>> +                echo "${file}"
>>>>>               done
>>>>>           done < "${include}"
>>>>>       done | sort -u
>>>

Adolf Belka March 29, 2022, 2:17 p.m. UTC | #8

Hi Michael,

On 29/03/2022 16:03, Michael Tremer wrote:
> Hey,
> 
>> On 29 Mar 2022, at 15:01, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>
>> Hi All,
>>
>> On 29/03/2022 15:36, Bernhard Bitsch wrote:
>>> Hi,
>>> Am 29.03.2022 um 15:11 schrieb Michael Tremer:
>>>> Hello,
>>>>
>>>> Could you please send the file listing to find out what is being included what shouldn’t?
>> Unfortunately, as I stopped the backup continuing once it had reached 1.2GB, the file created was not able to be opened. Probably stopping the backup corrupted it in some way.
>>>>
>>> Just for easy quick handling, you can use
>>> '/var/ipfire/backup/bin/backup.pl list' to show the files included.
>> I gave this a go and here is the output from the command.
>>
>> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
>> /
> 
> You have / on the list which will cause that the entire system is being put into the backup.
> 
> Do you have anything custom in include.user?
I have the following two lines:-

/home/ahb
/etc/sudoers

Regards,
Adolf
> 
>> /etc/conntrackd/conntrackd.conf
>> /etc/group
>> /etc/hosts
>> /etc/hosts.allow
>> /etc/hosts.deny
>> /etc/httpd/server.crt
>> /etc/httpd/server.csr
>> /etc/httpd/server-ecdsa.crt
>> /etc/httpd/server-ecdsa.csr
>> /etc/httpd/server-ecdsa.key
>> /etc/httpd/server.key
>> /etc/ipsec.user.conf
>> /etc/ipsec.user-post.conf
>> /etc/ipsec.user.secrets
>> /etc/logrotate.d
>> /etc/passwd
>> /etc/shadow
>> /etc/squid/squid.conf.local
>> /etc/squid/squid.conf.pre.local
>> /etc/ssh/sshd_config
>> /etc/ssh/ssh_host_ecdsa_key
>> /etc/ssh/ssh_host_ecdsa_key.pub
>> /etc/ssh/ssh_host_ed25519_key
>> /etc/ssh/ssh_host_ed25519_key.pub
>> /etc/ssh/ssh_host_rsa_key
>> /etc/ssh/ssh_host_rsa_key.pub
>> //etc/sudoers
>> /etc/sysconfig/createfiles
>> /etc/sysconfig/firewall.local
>> /etc/sysconfig/lm_sensors
>> /etc/sysconfig/modules
>> /etc/sysconfig/ramdisk
>> /etc/sysconfig/rc
>> /etc/sysconfig/rc.local
>> /etc/unbound
>> //home/ahb
>> /root/.bash_history
>> /root/.gitconfig
>> /root/.ssh
>> /var/ipfire/accounting/settings.conf
>> /var/ipfire/auth/users
>> /var/ipfire/backup/addons/backup
>> /var/ipfire/backup/exclude.user
>> /var/ipfire/backup/include.user
>> /var/ipfire/ca/cacert.pem
>> /var/ipfire/captive/agb.txt
>> /var/ipfire/captive/clients
>> /var/ipfire/captive/coupons
>> /var/ipfire/captive/logo.dat
>> /var/ipfire/captive/settings
>> /var/ipfire/captive/terms.txt
>> /var/ipfire/captive/voucher_out
>> /var/ipfire/certs/hostcert.pem
>> /var/ipfire/certs/hostkey.pem
>> /var/ipfire/certs/phoebevmipseccert.pem
>> /var/ipfire/connscheduler/connscheduler.conf
>> /var/ipfire/crls/cacrl.pem
>> /var/ipfire/cups/cups-browsed.conf
>> /var/ipfire/cups/subscriptions.conf
>> /var/ipfire/ddns/config
>> /var/ipfire/ddns/ddns.conf
>> /var/ipfire/ddns/settings
>> /var/ipfire/dhcp/advoptions
>> /var/ipfire/dhcp/advoptions-list
>> /var/ipfire/dhcpc/dhcpcd.conf
>> /var/ipfire/dhcp/dhcpd.conf
>> /var/ipfire/dhcp/dhcpd.conf.local
>> /var/ipfire/dhcp/enable_blue
>> /var/ipfire/dhcp/enable_green
>> /var/ipfire/dhcp/fixleases
>> /var/ipfire/dhcp/settings
>> /var/ipfire/dma/auth.conf
>> /var/ipfire/dma/dma.conf
>> /var/ipfire/dma/mail.conf
>> /var/ipfire/dns
>> /var/ipfire/dnsforward/config
>> /var/ipfire/dns/settings
>> /var/ipfire/ethernet/aliases
>> /var/ipfire/ethernet/settings
>> /var/ipfire/ethernet/wireless
>> /var/ipfire/extrahd/settings
>> /var/ipfire/firewall
>> /var/ipfire/firewall/config
>> /var/ipfire/firewall/settings
>> /var/ipfire/fwhosts
>> /var/ipfire/isdn/settings
>> /var/ipfire/logging/settings
>> /var/ipfire/mac/settings
>> /var/ipfire/main/firstsetup_ok
>> /var/ipfire/main/gpl_accepted
>> /var/ipfire/main/hostname.conf
>> /var/ipfire/main/hosts
>> /var/ipfire/main/manualpages
>> /var/ipfire/main/routing
>> /var/ipfire/main/security
>> /var/ipfire/main/send_profile
>> /var/ipfire/main/settings
>> /var/ipfire/modem/settings
>> /var/ipfire/optionsfw/settings
>> /var/ipfire/ovpn
>> /var/ipfire/ovpn/ccd.conf
>> /var/ipfire/ovpn/collectd.vpn
>> /var/ipfire/ovpn/enable
>> /var/ipfire/ovpn/server.conf
>> /var/ipfire/ovpn/settings
>> /var/ipfire/pakfire/settings
>> /var/ipfire/ppp
>> /var/ipfire/ppp/fake-resolv.conf
>> /var/ipfire/ppp/settings
>> /var/ipfire/private/cakey.pem
>> /var/ipfire/proxy
>> /var/ipfire/proxy/asnbl-helper.conf
>> /var/ipfire/proxy/cachemgr.conf
>> /var/ipfire/proxy/enable
>> /var/ipfire/proxy/settings
>> /var/ipfire/proxy/squid.conf
>> /var/ipfire/qos/bin
>> /var/ipfire/qos/bin/qos.sh
>> /var/ipfire/qos/classes
>> /var/ipfire/qos/level7config
>> /var/ipfire/qos/portconfig
>> /var/ipfire/qos/settings
>> /var/ipfire/qos/subclasses
>> /var/ipfire/qos/tosconfig
>> /var/ipfire/remote/enablessh
>> /var/ipfire/remote/settings
>> /var/ipfire/sensors/settings
>> /var/ipfire/suricata/oinkmaster.conf
>> /var/ipfire/suricata/oinkmaster-modify-sids.conf
>> /var/ipfire/suricata/oinkmaster-provider-includes.conf
>> /var/ipfire/suricata/providers-settings
>> /var/ipfire/suricata/settings
>> /var/ipfire/suricata/suricata-default-rules.yaml
>> /var/ipfire/suricata/suricata-dns-servers.yaml
>> /var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
>> /var/ipfire/suricata/suricata-homenet.yaml
>> /var/ipfire/suricata/suricata-http-ports.yaml
>> /var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml
>> /var/ipfire/suricata/suricata-used-providers.yaml
>> /var/ipfire/time/
>> /var/ipfire/time/counter.conf
>> /var/ipfire/time/enable
>> /var/ipfire/time/settime.conf
>> /var/ipfire/time/settings
>> /var/ipfire/upnp/settings
>> /var/ipfire/urlfilter
>> /var/ipfire/urlfilter/settings
>> /var/ipfire/urlfilter/squidGuard.conf
>> /var/ipfire/vpn
>> /var/ipfire/vpn/config
>> /var/ipfire/vpn/ipsec.conf
>> /var/ipfire/vpn/settings
>> /var/ipfire/wakeonlan/clients.conf
>> /var/ipfire/wio/wio.conf
>> /var/ipfire/wireless/config
>> /var/ipfire/wireless/settings
>> /var/lib/suricata
>> /var/log/rrd/collectd
>> /var/log/rrd/hddshutdown-md127.rrd
>> /var/log/rrd/hddshutdown-sda.rrd
>> /var/log/rrd/hddshutdown-sdb.rrd
>> /var/log/rrd/hddtemp-md127.rrd
>> /var/log/rrd/hddtemp-sda.rrd
>> /var/log/rrd/hddtemp-sdb.rrd
>> /var/log/rrd/wio
>> /var/log/vnstat
>> /var/tmp/idsrules-emerging.tar.gz
>> /var/tmp/idsrules-sslbl_blacklist.rules
>>
>> Regards,
>> Adolf.
>>> Bernhard
>>>> -Michael
>>>>
>>>>> On 29 Mar 2022, at 14:10, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>>>>
>>>>> Hi Michael,
>>>>>
>>>>> Just tried this patch out on my vm testbed system and it still doesn't work for me. The backup file had got to 1.3GB when I deleted the backup file as it was still growing. The normal correct backup file on that vm machine is around 7MB
>>>>>
>>>>> The above was the case for both running it from the WUI or from the command line from my unprivileged user using sudo backupctrl exclude
>>>>>
>>>>> Regards,
>>>>>
>>>>> Adolf.
>>>>>
>>>>>
>>>>> On 29/03/2022 14:27, Michael Tremer wrote:
>>>>>> This patch fixes globbing expansion in the backup include file list
>>>>>> which got broken in c7e0d73e7cfd7be95db9d0a5f3392b8241813d5b.
>>>>>>
>>>>>> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
>>>>>> ---
>>>>>>    config/backup/backup.pl | 8 ++++----
>>>>>>    1 file changed, 4 insertions(+), 4 deletions(-)
>>>>>>
>>>>>> diff --git a/config/backup/backup.pl b/config/backup/backup.pl
>>>>>> index a2337cf23..6f9295e94 100644
>>>>>> --- a/config/backup/backup.pl
>>>>>> +++ b/config/backup/backup.pl
>>>>>> @@ -19,6 +19,8 @@
>>>>>>    #                                                                             #
>>>>>>    ###############################################################################
>>>>>>    +shopt -s nullglob
>>>>>> +
>>>>>>    NOW="$(date "+%Y-%m-%d-%H:%M")"
>>>>>>      list_addons() {
>>>>>> @@ -38,10 +40,8 @@ process_includes() {
>>>>>>        for include in $@; do
>>>>>>            local file
>>>>>>            while read -r file; do
>>>>>> -            for file in ${file}; do
>>>>>> -                if [ -e "/${file}" ]; then
>>>>>> -                    echo "${file}"
>>>>>> -                fi
>>>>>> +            for file in /${file}; do
>>>>>> +                echo "${file}"
>>>>>>                done
>>>>>>            done < "${include}"
>>>>>>        done | sort -u
>>>>
>

Bernhard Bitsch March 29, 2022, 2:22 p.m. UTC | #9

Looks like your include.user file contains absolute paths.

Am 29.03.2022 um 16:14 schrieb Adolf Belka:
> Hi All,
> 
> This is what I get with the pushd/popd version in place on the same system
> 
> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
> etc/group
> etc/hosts
> etc/hosts.allow
> etc/hosts.deny
> etc/httpd/server.crt
> etc/httpd/server.csr
> etc/httpd/server-ecdsa.crt
> etc/httpd/server-ecdsa.csr
> etc/httpd/server-ecdsa.key
> etc/httpd/server.key
> etc/ipsec.user.conf
> etc/ipsec.user.secrets
> etc/logrotate.d
> etc/passwd
> etc/shadow
> etc/ssh/sshd_config
> etc/ssh/ssh_host_ecdsa_key
> etc/ssh/ssh_host_ecdsa_key.pub
> etc/ssh/ssh_host_ed25519_key
> etc/ssh/ssh_host_ed25519_key.pub
> etc/ssh/ssh_host_rsa_key
> etc/ssh/ssh_host_rsa_key.pub
> /etc/sudoers
> etc/sysconfig/createfiles
> etc/sysconfig/firewall.local
> etc/sysconfig/lm_sensors
> etc/sysconfig/modules
> etc/sysconfig/ramdisk
> etc/sysconfig/rc
> etc/sysconfig/rc.local
> etc/unbound
> /home/ahb
> root/.bash_history
> var/ipfire/accounting/settings.conf
> var/ipfire/auth/users
> var/ipfire/backup/addons/backup
> var/ipfire/backup/exclude.user
> var/ipfire/backup/include.user
> var/ipfire/ca/cacert.pem
> var/ipfire/captive/agb.txt
> var/ipfire/captive/clients
> var/ipfire/captive/coupons
> var/ipfire/captive/logo.dat
> var/ipfire/captive/settings
> var/ipfire/captive/terms.txt
> var/ipfire/captive/voucher_out
> var/ipfire/certs/hostcert.pem
> var/ipfire/certs/hostkey.pem
> var/ipfire/certs/phoebevmipseccert.pem
> var/ipfire/connscheduler/connscheduler.conf
> var/ipfire/crls/cacrl.pem
> var/ipfire/cups/cups-browsed.conf
> var/ipfire/cups/subscriptions.conf
> var/ipfire/ddns/config
> var/ipfire/ddns/ddns.conf
> var/ipfire/ddns/settings
> var/ipfire/dhcp/advoptions
> var/ipfire/dhcp/advoptions-list
> var/ipfire/dhcpc/dhcpcd.conf
> var/ipfire/dhcp/dhcpd.conf
> var/ipfire/dhcp/dhcpd.conf.local
> var/ipfire/dhcp/enable_blue
> var/ipfire/dhcp/enable_green
> var/ipfire/dhcp/fixleases
> var/ipfire/dhcp/settings
> var/ipfire/dma/auth.conf
> var/ipfire/dma/dma.conf
> var/ipfire/dma/mail.conf
> var/ipfire/dns
> var/ipfire/dnsforward/config
> var/ipfire/dns/settings
> var/ipfire/ethernet/aliases
> var/ipfire/ethernet/settings
> var/ipfire/ethernet/wireless
> var/ipfire/extrahd/settings
> var/ipfire/firewall
> var/ipfire/firewall/config
> var/ipfire/firewall/settings
> var/ipfire/fwhosts
> var/ipfire/isdn/settings
> var/ipfire/logging/settings
> var/ipfire/mac/settings
> var/ipfire/main/firstsetup_ok
> var/ipfire/main/gpl_accepted
> var/ipfire/main/hostname.conf
> var/ipfire/main/hosts
> var/ipfire/main/manualpages
> var/ipfire/main/routing
> var/ipfire/main/security
> var/ipfire/main/send_profile
> var/ipfire/main/settings
> var/ipfire/modem/settings
> var/ipfire/optionsfw/settings
> var/ipfire/ovpn
> var/ipfire/ovpn/ccd.conf
> var/ipfire/ovpn/collectd.vpn
> var/ipfire/ovpn/enable
> var/ipfire/ovpn/server.conf
> var/ipfire/ovpn/settings
> var/ipfire/pakfire/settings
> var/ipfire/ppp
> var/ipfire/ppp/fake-resolv.conf
> var/ipfire/ppp/settings
> var/ipfire/private/cakey.pem
> var/ipfire/proxy
> var/ipfire/proxy/asnbl-helper.conf
> var/ipfire/proxy/cachemgr.conf
> var/ipfire/proxy/enable
> var/ipfire/proxy/settings
> var/ipfire/proxy/squid.conf
> var/ipfire/qos/bin
> var/ipfire/qos/classes
> var/ipfire/qos/level7config
> var/ipfire/qos/portconfig
> var/ipfire/qos/settings
> var/ipfire/qos/subclasses
> var/ipfire/qos/tosconfig
> var/ipfire/remote/enablessh
> var/ipfire/remote/settings
> var/ipfire/sensors/settings
> var/ipfire/suricata/oinkmaster.conf
> var/ipfire/suricata/oinkmaster-modify-sids.conf
> var/ipfire/suricata/oinkmaster-provider-includes.conf
> var/ipfire/suricata/providers-settings
> var/ipfire/suricata/settings
> var/ipfire/suricata/suricata-default-rules.yaml
> var/ipfire/suricata/suricata-dns-servers.yaml
> var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
> var/ipfire/suricata/suricata-homenet.yaml
> var/ipfire/suricata/suricata-http-ports.yaml
> var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml
> var/ipfire/suricata/suricata-used-providers.yaml
> var/ipfire/time/
> var/ipfire/time/counter.conf
> var/ipfire/time/enable
> var/ipfire/time/settime.conf
> var/ipfire/time/settings
> var/ipfire/upnp/settings
> var/ipfire/urlfilter
> var/ipfire/urlfilter/settings
> var/ipfire/urlfilter/squidGuard.conf
> var/ipfire/vpn
> var/ipfire/vpn/config
> var/ipfire/vpn/ipsec.conf
> var/ipfire/vpn/settings
> var/ipfire/wakeonlan/clients.conf
> var/ipfire/wio/wio.conf
> var/ipfire/wireless/config
> var/ipfire/wireless/settings
> var/lib/suricata
> var/log/rrd/collectd
> var/log/rrd/hddshutdown-md127.rrd
> var/log/rrd/hddshutdown-sda.rrd
> var/log/rrd/hddshutdown-sdb.rrd
> var/log/rrd/hddtemp-md127.rrd
> var/log/rrd/hddtemp-sda.rrd
> var/log/rrd/hddtemp-sdb.rrd
> var/log/rrd/wio
> var/log/vnstat
> var/tmp/idsrules-emerging.tar.gz
> var/tmp/idsrules-sslbl_blacklist.rules
> 
> The following are in the previous list but not in this one:-
> 
> /
> /etc/conntrackd/conntrackd.conf
> /etc/ipsec.user-post.conf
> /root/.gitconfig
> /root/.ssh
> 
> Regards,
> 
> Adolf.
> 
> On 29/03/2022 16:01, Adolf Belka wrote:
>> Hi All,
>>
>> On 29/03/2022 15:36, Bernhard Bitsch wrote:
>>> Hi,
>>>
>>> Am 29.03.2022 um 15:11 schrieb Michael Tremer:
>>>> Hello,
>>>>
>>>> Could you please send the file listing to find out what is being 
>>>> included what shouldn’t?
>> Unfortunately, as I stopped the backup continuing once it had reached 
>> 1.2GB, the file created was not able to be opened. Probably stopping 
>> the backup corrupted it in some way.
>>>>
>>>
>>> Just for easy quick handling, you can use
>>> '/var/ipfire/backup/bin/backup.pl list' to show the files included.
>> I gave this a go and here is the output from the command.
>>
>> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
>> /
>> /etc/conntrackd/conntrackd.conf
>> /etc/group
>> /etc/hosts
>> /etc/hosts.allow
>> /etc/hosts.deny
>> /etc/httpd/server.crt
>> /etc/httpd/server.csr
>> /etc/httpd/server-ecdsa.crt
>> /etc/httpd/server-ecdsa.csr
>> /etc/httpd/server-ecdsa.key
>> /etc/httpd/server.key
>> /etc/ipsec.user.conf
>> /etc/ipsec.user-post.conf
>> /etc/ipsec.user.secrets
>> /etc/logrotate.d
>> /etc/passwd
>> /etc/shadow
>> /etc/squid/squid.conf.local
>> /etc/squid/squid.conf.pre.local
>> /etc/ssh/sshd_config
>> /etc/ssh/ssh_host_ecdsa_key
>> /etc/ssh/ssh_host_ecdsa_key.pub
>> /etc/ssh/ssh_host_ed25519_key
>> /etc/ssh/ssh_host_ed25519_key.pub
>> /etc/ssh/ssh_host_rsa_key
>> /etc/ssh/ssh_host_rsa_key.pub
>> //etc/sudoers
>> /etc/sysconfig/createfiles
>> /etc/sysconfig/firewall.local
>> /etc/sysconfig/lm_sensors
>> /etc/sysconfig/modules
>> /etc/sysconfig/ramdisk
>> /etc/sysconfig/rc
>> /etc/sysconfig/rc.local
>> /etc/unbound
>> //home/ahb
>> /root/.bash_history
>> /root/.gitconfig
>> /root/.ssh
>> /var/ipfire/accounting/settings.conf
>> /var/ipfire/auth/users
>> /var/ipfire/backup/addons/backup
>> /var/ipfire/backup/exclude.user
>> /var/ipfire/backup/include.user
>> /var/ipfire/ca/cacert.pem
>> /var/ipfire/captive/agb.txt
>> /var/ipfire/captive/clients
>> /var/ipfire/captive/coupons
>> /var/ipfire/captive/logo.dat
>> /var/ipfire/captive/settings
>> /var/ipfire/captive/terms.txt
>> /var/ipfire/captive/voucher_out
>> /var/ipfire/certs/hostcert.pem
>> /var/ipfire/certs/hostkey.pem
>> /var/ipfire/certs/phoebevmipseccert.pem
>> /var/ipfire/connscheduler/connscheduler.conf
>> /var/ipfire/crls/cacrl.pem
>> /var/ipfire/cups/cups-browsed.conf
>> /var/ipfire/cups/subscriptions.conf
>> /var/ipfire/ddns/config
>> /var/ipfire/ddns/ddns.conf
>> /var/ipfire/ddns/settings
>> /var/ipfire/dhcp/advoptions
>> /var/ipfire/dhcp/advoptions-list
>> /var/ipfire/dhcpc/dhcpcd.conf
>> /var/ipfire/dhcp/dhcpd.conf
>> /var/ipfire/dhcp/dhcpd.conf.local
>> /var/ipfire/dhcp/enable_blue
>> /var/ipfire/dhcp/enable_green
>> /var/ipfire/dhcp/fixleases
>> /var/ipfire/dhcp/settings
>> /var/ipfire/dma/auth.conf
>> /var/ipfire/dma/dma.conf
>> /var/ipfire/dma/mail.conf
>> /var/ipfire/dns
>> /var/ipfire/dnsforward/config
>> /var/ipfire/dns/settings
>> /var/ipfire/ethernet/aliases
>> /var/ipfire/ethernet/settings
>> /var/ipfire/ethernet/wireless
>> /var/ipfire/extrahd/settings
>> /var/ipfire/firewall
>> /var/ipfire/firewall/config
>> /var/ipfire/firewall/settings
>> /var/ipfire/fwhosts
>> /var/ipfire/isdn/settings
>> /var/ipfire/logging/settings
>> /var/ipfire/mac/settings
>> /var/ipfire/main/firstsetup_ok
>> /var/ipfire/main/gpl_accepted
>> /var/ipfire/main/hostname.conf
>> /var/ipfire/main/hosts
>> /var/ipfire/main/manualpages
>> /var/ipfire/main/routing
>> /var/ipfire/main/security
>> /var/ipfire/main/send_profile
>> /var/ipfire/main/settings
>> /var/ipfire/modem/settings
>> /var/ipfire/optionsfw/settings
>> /var/ipfire/ovpn
>> /var/ipfire/ovpn/ccd.conf
>> /var/ipfire/ovpn/collectd.vpn
>> /var/ipfire/ovpn/enable
>> /var/ipfire/ovpn/server.conf
>> /var/ipfire/ovpn/settings
>> /var/ipfire/pakfire/settings
>> /var/ipfire/ppp
>> /var/ipfire/ppp/fake-resolv.conf
>> /var/ipfire/ppp/settings
>> /var/ipfire/private/cakey.pem
>> /var/ipfire/proxy
>> /var/ipfire/proxy/asnbl-helper.conf
>> /var/ipfire/proxy/cachemgr.conf
>> /var/ipfire/proxy/enable
>> /var/ipfire/proxy/settings
>> /var/ipfire/proxy/squid.conf
>> /var/ipfire/qos/bin
>> /var/ipfire/qos/bin/qos.sh
>> /var/ipfire/qos/classes
>> /var/ipfire/qos/level7config
>> /var/ipfire/qos/portconfig
>> /var/ipfire/qos/settings
>> /var/ipfire/qos/subclasses
>> /var/ipfire/qos/tosconfig
>> /var/ipfire/remote/enablessh
>> /var/ipfire/remote/settings
>> /var/ipfire/sensors/settings
>> /var/ipfire/suricata/oinkmaster.conf
>> /var/ipfire/suricata/oinkmaster-modify-sids.conf
>> /var/ipfire/suricata/oinkmaster-provider-includes.conf
>> /var/ipfire/suricata/providers-settings
>> /var/ipfire/suricata/settings
>> /var/ipfire/suricata/suricata-default-rules.yaml
>> /var/ipfire/suricata/suricata-dns-servers.yaml
>> /var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
>> /var/ipfire/suricata/suricata-homenet.yaml
>> /var/ipfire/suricata/suricata-http-ports.yaml
>> /var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml
>> /var/ipfire/suricata/suricata-used-providers.yaml
>> /var/ipfire/time/
>> /var/ipfire/time/counter.conf
>> /var/ipfire/time/enable
>> /var/ipfire/time/settime.conf
>> /var/ipfire/time/settings
>> /var/ipfire/upnp/settings
>> /var/ipfire/urlfilter
>> /var/ipfire/urlfilter/settings
>> /var/ipfire/urlfilter/squidGuard.conf
>> /var/ipfire/vpn
>> /var/ipfire/vpn/config
>> /var/ipfire/vpn/ipsec.conf
>> /var/ipfire/vpn/settings
>> /var/ipfire/wakeonlan/clients.conf
>> /var/ipfire/wio/wio.conf
>> /var/ipfire/wireless/config
>> /var/ipfire/wireless/settings
>> /var/lib/suricata
>> /var/log/rrd/collectd
>> /var/log/rrd/hddshutdown-md127.rrd
>> /var/log/rrd/hddshutdown-sda.rrd
>> /var/log/rrd/hddshutdown-sdb.rrd
>> /var/log/rrd/hddtemp-md127.rrd
>> /var/log/rrd/hddtemp-sda.rrd
>> /var/log/rrd/hddtemp-sdb.rrd
>> /var/log/rrd/wio
>> /var/log/vnstat
>> /var/tmp/idsrules-emerging.tar.gz
>> /var/tmp/idsrules-sslbl_blacklist.rules
>>
>> Regards,
>> Adolf.
>>>
>>> Bernhard
>>>> -Michael
>>>>
>>>>> On 29 Mar 2022, at 14:10, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>>>>
>>>>> Hi Michael,
>>>>>
>>>>> Just tried this patch out on my vm testbed system and it still 
>>>>> doesn't work for me. The backup file had got to 1.3GB when I 
>>>>> deleted the backup file as it was still growing. The normal correct 
>>>>> backup file on that vm machine is around 7MB
>>>>>
>>>>> The above was the case for both running it from the WUI or from the 
>>>>> command line from my unprivileged user using sudo backupctrl exclude
>>>>>
>>>>> Regards,
>>>>>
>>>>> Adolf.
>>>>>
>>>>>
>>>>> On 29/03/2022 14:27, Michael Tremer wrote:
>>>>>> This patch fixes globbing expansion in the backup include file list
>>>>>> which got broken in c7e0d73e7cfd7be95db9d0a5f3392b8241813d5b.
>>>>>>
>>>>>> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
>>>>>> ---
>>>>>>   config/backup/backup.pl | 8 ++++----
>>>>>>   1 file changed, 4 insertions(+), 4 deletions(-)
>>>>>>
>>>>>> diff --git a/config/backup/backup.pl b/config/backup/backup.pl
>>>>>> index a2337cf23..6f9295e94 100644
>>>>>> --- a/config/backup/backup.pl
>>>>>> +++ b/config/backup/backup.pl
>>>>>> @@ -19,6 +19,8 @@
>>>>>> # #
>>>>>> ############################################################################### 
>>>>>>
>>>>>>   +shopt -s nullglob
>>>>>> +
>>>>>>   NOW="$(date "+%Y-%m-%d-%H:%M")"
>>>>>>     list_addons() {
>>>>>> @@ -38,10 +40,8 @@ process_includes() {
>>>>>>       for include in $@; do
>>>>>>           local file
>>>>>>           while read -r file; do
>>>>>> -            for file in ${file}; do
>>>>>> -                if [ -e "/${file}" ]; then
>>>>>> -                    echo "${file}"
>>>>>> -                fi
>>>>>> +            for file in /${file}; do
>>>>>> +                echo "${file}"
>>>>>>               done
>>>>>>           done < "${include}"
>>>>>>       done | sort -u
>>>>

Michael Tremer March 29, 2022, 2:22 p.m. UTC | #10

> On 29 Mar 2022, at 15:22, Bernhard Bitsch <bbitsch@ipfire.org> wrote:
> 
> Looks like your include.user file contains absolute paths.

Or an empty line maybe?

> 
> Am 29.03.2022 um 16:14 schrieb Adolf Belka:
>> Hi All,
>> This is what I get with the pushd/popd version in place on the same system
>> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
>> etc/group
>> etc/hosts
>> etc/hosts.allow
>> etc/hosts.deny
>> etc/httpd/server.crt
>> etc/httpd/server.csr
>> etc/httpd/server-ecdsa.crt
>> etc/httpd/server-ecdsa.csr
>> etc/httpd/server-ecdsa.key
>> etc/httpd/server.key
>> etc/ipsec.user.conf
>> etc/ipsec.user.secrets
>> etc/logrotate.d
>> etc/passwd
>> etc/shadow
>> etc/ssh/sshd_config
>> etc/ssh/ssh_host_ecdsa_key
>> etc/ssh/ssh_host_ecdsa_key.pub
>> etc/ssh/ssh_host_ed25519_key
>> etc/ssh/ssh_host_ed25519_key.pub
>> etc/ssh/ssh_host_rsa_key
>> etc/ssh/ssh_host_rsa_key.pub
>> /etc/sudoers
>> etc/sysconfig/createfiles
>> etc/sysconfig/firewall.local
>> etc/sysconfig/lm_sensors
>> etc/sysconfig/modules
>> etc/sysconfig/ramdisk
>> etc/sysconfig/rc
>> etc/sysconfig/rc.local
>> etc/unbound
>> /home/ahb
>> root/.bash_history
>> var/ipfire/accounting/settings.conf
>> var/ipfire/auth/users
>> var/ipfire/backup/addons/backup
>> var/ipfire/backup/exclude.user
>> var/ipfire/backup/include.user
>> var/ipfire/ca/cacert.pem
>> var/ipfire/captive/agb.txt
>> var/ipfire/captive/clients
>> var/ipfire/captive/coupons
>> var/ipfire/captive/logo.dat
>> var/ipfire/captive/settings
>> var/ipfire/captive/terms.txt
>> var/ipfire/captive/voucher_out
>> var/ipfire/certs/hostcert.pem
>> var/ipfire/certs/hostkey.pem
>> var/ipfire/certs/phoebevmipseccert.pem
>> var/ipfire/connscheduler/connscheduler.conf
>> var/ipfire/crls/cacrl.pem
>> var/ipfire/cups/cups-browsed.conf
>> var/ipfire/cups/subscriptions.conf
>> var/ipfire/ddns/config
>> var/ipfire/ddns/ddns.conf
>> var/ipfire/ddns/settings
>> var/ipfire/dhcp/advoptions
>> var/ipfire/dhcp/advoptions-list
>> var/ipfire/dhcpc/dhcpcd.conf
>> var/ipfire/dhcp/dhcpd.conf
>> var/ipfire/dhcp/dhcpd.conf.local
>> var/ipfire/dhcp/enable_blue
>> var/ipfire/dhcp/enable_green
>> var/ipfire/dhcp/fixleases
>> var/ipfire/dhcp/settings
>> var/ipfire/dma/auth.conf
>> var/ipfire/dma/dma.conf
>> var/ipfire/dma/mail.conf
>> var/ipfire/dns
>> var/ipfire/dnsforward/config
>> var/ipfire/dns/settings
>> var/ipfire/ethernet/aliases
>> var/ipfire/ethernet/settings
>> var/ipfire/ethernet/wireless
>> var/ipfire/extrahd/settings
>> var/ipfire/firewall
>> var/ipfire/firewall/config
>> var/ipfire/firewall/settings
>> var/ipfire/fwhosts
>> var/ipfire/isdn/settings
>> var/ipfire/logging/settings
>> var/ipfire/mac/settings
>> var/ipfire/main/firstsetup_ok
>> var/ipfire/main/gpl_accepted
>> var/ipfire/main/hostname.conf
>> var/ipfire/main/hosts
>> var/ipfire/main/manualpages
>> var/ipfire/main/routing
>> var/ipfire/main/security
>> var/ipfire/main/send_profile
>> var/ipfire/main/settings
>> var/ipfire/modem/settings
>> var/ipfire/optionsfw/settings
>> var/ipfire/ovpn
>> var/ipfire/ovpn/ccd.conf
>> var/ipfire/ovpn/collectd.vpn
>> var/ipfire/ovpn/enable
>> var/ipfire/ovpn/server.conf
>> var/ipfire/ovpn/settings
>> var/ipfire/pakfire/settings
>> var/ipfire/ppp
>> var/ipfire/ppp/fake-resolv.conf
>> var/ipfire/ppp/settings
>> var/ipfire/private/cakey.pem
>> var/ipfire/proxy
>> var/ipfire/proxy/asnbl-helper.conf
>> var/ipfire/proxy/cachemgr.conf
>> var/ipfire/proxy/enable
>> var/ipfire/proxy/settings
>> var/ipfire/proxy/squid.conf
>> var/ipfire/qos/bin
>> var/ipfire/qos/classes
>> var/ipfire/qos/level7config
>> var/ipfire/qos/portconfig
>> var/ipfire/qos/settings
>> var/ipfire/qos/subclasses
>> var/ipfire/qos/tosconfig
>> var/ipfire/remote/enablessh
>> var/ipfire/remote/settings
>> var/ipfire/sensors/settings
>> var/ipfire/suricata/oinkmaster.conf
>> var/ipfire/suricata/oinkmaster-modify-sids.conf
>> var/ipfire/suricata/oinkmaster-provider-includes.conf
>> var/ipfire/suricata/providers-settings
>> var/ipfire/suricata/settings
>> var/ipfire/suricata/suricata-default-rules.yaml
>> var/ipfire/suricata/suricata-dns-servers.yaml
>> var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
>> var/ipfire/suricata/suricata-homenet.yaml
>> var/ipfire/suricata/suricata-http-ports.yaml
>> var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml
>> var/ipfire/suricata/suricata-used-providers.yaml
>> var/ipfire/time/
>> var/ipfire/time/counter.conf
>> var/ipfire/time/enable
>> var/ipfire/time/settime.conf
>> var/ipfire/time/settings
>> var/ipfire/upnp/settings
>> var/ipfire/urlfilter
>> var/ipfire/urlfilter/settings
>> var/ipfire/urlfilter/squidGuard.conf
>> var/ipfire/vpn
>> var/ipfire/vpn/config
>> var/ipfire/vpn/ipsec.conf
>> var/ipfire/vpn/settings
>> var/ipfire/wakeonlan/clients.conf
>> var/ipfire/wio/wio.conf
>> var/ipfire/wireless/config
>> var/ipfire/wireless/settings
>> var/lib/suricata
>> var/log/rrd/collectd
>> var/log/rrd/hddshutdown-md127.rrd
>> var/log/rrd/hddshutdown-sda.rrd
>> var/log/rrd/hddshutdown-sdb.rrd
>> var/log/rrd/hddtemp-md127.rrd
>> var/log/rrd/hddtemp-sda.rrd
>> var/log/rrd/hddtemp-sdb.rrd
>> var/log/rrd/wio
>> var/log/vnstat
>> var/tmp/idsrules-emerging.tar.gz
>> var/tmp/idsrules-sslbl_blacklist.rules
>> The following are in the previous list but not in this one:-
>> /
>> /etc/conntrackd/conntrackd.conf
>> /etc/ipsec.user-post.conf
>> /root/.gitconfig
>> /root/.ssh
>> Regards,
>> Adolf.
>> On 29/03/2022 16:01, Adolf Belka wrote:
>>> Hi All,
>>> 
>>> On 29/03/2022 15:36, Bernhard Bitsch wrote:
>>>> Hi,
>>>> 
>>>> Am 29.03.2022 um 15:11 schrieb Michael Tremer:
>>>>> Hello,
>>>>> 
>>>>> Could you please send the file listing to find out what is being included what shouldn’t?
>>> Unfortunately, as I stopped the backup continuing once it had reached 1.2GB, the file created was not able to be opened. Probably stopping the backup corrupted it in some way.
>>>>> 
>>>> 
>>>> Just for easy quick handling, you can use
>>>> '/var/ipfire/backup/bin/backup.pl list' to show the files included.
>>> I gave this a go and here is the output from the command.
>>> 
>>> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
>>> /
>>> /etc/conntrackd/conntrackd.conf
>>> /etc/group
>>> /etc/hosts
>>> /etc/hosts.allow
>>> /etc/hosts.deny
>>> /etc/httpd/server.crt
>>> /etc/httpd/server.csr
>>> /etc/httpd/server-ecdsa.crt
>>> /etc/httpd/server-ecdsa.csr
>>> /etc/httpd/server-ecdsa.key
>>> /etc/httpd/server.key
>>> /etc/ipsec.user.conf
>>> /etc/ipsec.user-post.conf
>>> /etc/ipsec.user.secrets
>>> /etc/logrotate.d
>>> /etc/passwd
>>> /etc/shadow
>>> /etc/squid/squid.conf.local
>>> /etc/squid/squid.conf.pre.local
>>> /etc/ssh/sshd_config
>>> /etc/ssh/ssh_host_ecdsa_key
>>> /etc/ssh/ssh_host_ecdsa_key.pub
>>> /etc/ssh/ssh_host_ed25519_key
>>> /etc/ssh/ssh_host_ed25519_key.pub
>>> /etc/ssh/ssh_host_rsa_key
>>> /etc/ssh/ssh_host_rsa_key.pub
>>> //etc/sudoers
>>> /etc/sysconfig/createfiles
>>> /etc/sysconfig/firewall.local
>>> /etc/sysconfig/lm_sensors
>>> /etc/sysconfig/modules
>>> /etc/sysconfig/ramdisk
>>> /etc/sysconfig/rc
>>> /etc/sysconfig/rc.local
>>> /etc/unbound
>>> //home/ahb
>>> /root/.bash_history
>>> /root/.gitconfig
>>> /root/.ssh
>>> /var/ipfire/accounting/settings.conf
>>> /var/ipfire/auth/users
>>> /var/ipfire/backup/addons/backup
>>> /var/ipfire/backup/exclude.user
>>> /var/ipfire/backup/include.user
>>> /var/ipfire/ca/cacert.pem
>>> /var/ipfire/captive/agb.txt
>>> /var/ipfire/captive/clients
>>> /var/ipfire/captive/coupons
>>> /var/ipfire/captive/logo.dat
>>> /var/ipfire/captive/settings
>>> /var/ipfire/captive/terms.txt
>>> /var/ipfire/captive/voucher_out
>>> /var/ipfire/certs/hostcert.pem
>>> /var/ipfire/certs/hostkey.pem
>>> /var/ipfire/certs/phoebevmipseccert.pem
>>> /var/ipfire/connscheduler/connscheduler.conf
>>> /var/ipfire/crls/cacrl.pem
>>> /var/ipfire/cups/cups-browsed.conf
>>> /var/ipfire/cups/subscriptions.conf
>>> /var/ipfire/ddns/config
>>> /var/ipfire/ddns/ddns.conf
>>> /var/ipfire/ddns/settings
>>> /var/ipfire/dhcp/advoptions
>>> /var/ipfire/dhcp/advoptions-list
>>> /var/ipfire/dhcpc/dhcpcd.conf
>>> /var/ipfire/dhcp/dhcpd.conf
>>> /var/ipfire/dhcp/dhcpd.conf.local
>>> /var/ipfire/dhcp/enable_blue
>>> /var/ipfire/dhcp/enable_green
>>> /var/ipfire/dhcp/fixleases
>>> /var/ipfire/dhcp/settings
>>> /var/ipfire/dma/auth.conf
>>> /var/ipfire/dma/dma.conf
>>> /var/ipfire/dma/mail.conf
>>> /var/ipfire/dns
>>> /var/ipfire/dnsforward/config
>>> /var/ipfire/dns/settings
>>> /var/ipfire/ethernet/aliases
>>> /var/ipfire/ethernet/settings
>>> /var/ipfire/ethernet/wireless
>>> /var/ipfire/extrahd/settings
>>> /var/ipfire/firewall
>>> /var/ipfire/firewall/config
>>> /var/ipfire/firewall/settings
>>> /var/ipfire/fwhosts
>>> /var/ipfire/isdn/settings
>>> /var/ipfire/logging/settings
>>> /var/ipfire/mac/settings
>>> /var/ipfire/main/firstsetup_ok
>>> /var/ipfire/main/gpl_accepted
>>> /var/ipfire/main/hostname.conf
>>> /var/ipfire/main/hosts
>>> /var/ipfire/main/manualpages
>>> /var/ipfire/main/routing
>>> /var/ipfire/main/security
>>> /var/ipfire/main/send_profile
>>> /var/ipfire/main/settings
>>> /var/ipfire/modem/settings
>>> /var/ipfire/optionsfw/settings
>>> /var/ipfire/ovpn
>>> /var/ipfire/ovpn/ccd.conf
>>> /var/ipfire/ovpn/collectd.vpn
>>> /var/ipfire/ovpn/enable
>>> /var/ipfire/ovpn/server.conf
>>> /var/ipfire/ovpn/settings
>>> /var/ipfire/pakfire/settings
>>> /var/ipfire/ppp
>>> /var/ipfire/ppp/fake-resolv.conf
>>> /var/ipfire/ppp/settings
>>> /var/ipfire/private/cakey.pem
>>> /var/ipfire/proxy
>>> /var/ipfire/proxy/asnbl-helper.conf
>>> /var/ipfire/proxy/cachemgr.conf
>>> /var/ipfire/proxy/enable
>>> /var/ipfire/proxy/settings
>>> /var/ipfire/proxy/squid.conf
>>> /var/ipfire/qos/bin
>>> /var/ipfire/qos/bin/qos.sh
>>> /var/ipfire/qos/classes
>>> /var/ipfire/qos/level7config
>>> /var/ipfire/qos/portconfig
>>> /var/ipfire/qos/settings
>>> /var/ipfire/qos/subclasses
>>> /var/ipfire/qos/tosconfig
>>> /var/ipfire/remote/enablessh
>>> /var/ipfire/remote/settings
>>> /var/ipfire/sensors/settings
>>> /var/ipfire/suricata/oinkmaster.conf
>>> /var/ipfire/suricata/oinkmaster-modify-sids.conf
>>> /var/ipfire/suricata/oinkmaster-provider-includes.conf
>>> /var/ipfire/suricata/providers-settings
>>> /var/ipfire/suricata/settings
>>> /var/ipfire/suricata/suricata-default-rules.yaml
>>> /var/ipfire/suricata/suricata-dns-servers.yaml
>>> /var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
>>> /var/ipfire/suricata/suricata-homenet.yaml
>>> /var/ipfire/suricata/suricata-http-ports.yaml
>>> /var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml
>>> /var/ipfire/suricata/suricata-used-providers.yaml
>>> /var/ipfire/time/
>>> /var/ipfire/time/counter.conf
>>> /var/ipfire/time/enable
>>> /var/ipfire/time/settime.conf
>>> /var/ipfire/time/settings
>>> /var/ipfire/upnp/settings
>>> /var/ipfire/urlfilter
>>> /var/ipfire/urlfilter/settings
>>> /var/ipfire/urlfilter/squidGuard.conf
>>> /var/ipfire/vpn
>>> /var/ipfire/vpn/config
>>> /var/ipfire/vpn/ipsec.conf
>>> /var/ipfire/vpn/settings
>>> /var/ipfire/wakeonlan/clients.conf
>>> /var/ipfire/wio/wio.conf
>>> /var/ipfire/wireless/config
>>> /var/ipfire/wireless/settings
>>> /var/lib/suricata
>>> /var/log/rrd/collectd
>>> /var/log/rrd/hddshutdown-md127.rrd
>>> /var/log/rrd/hddshutdown-sda.rrd
>>> /var/log/rrd/hddshutdown-sdb.rrd
>>> /var/log/rrd/hddtemp-md127.rrd
>>> /var/log/rrd/hddtemp-sda.rrd
>>> /var/log/rrd/hddtemp-sdb.rrd
>>> /var/log/rrd/wio
>>> /var/log/vnstat
>>> /var/tmp/idsrules-emerging.tar.gz
>>> /var/tmp/idsrules-sslbl_blacklist.rules
>>> 
>>> Regards,
>>> Adolf.
>>>> 
>>>> Bernhard
>>>>> -Michael
>>>>> 
>>>>>> On 29 Mar 2022, at 14:10, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>>>>> 
>>>>>> Hi Michael,
>>>>>> 
>>>>>> Just tried this patch out on my vm testbed system and it still doesn't work for me. The backup file had got to 1.3GB when I deleted the backup file as it was still growing. The normal correct backup file on that vm machine is around 7MB
>>>>>> 
>>>>>> The above was the case for both running it from the WUI or from the command line from my unprivileged user using sudo backupctrl exclude
>>>>>> 
>>>>>> Regards,
>>>>>> 
>>>>>> Adolf.
>>>>>> 
>>>>>> 
>>>>>> On 29/03/2022 14:27, Michael Tremer wrote:
>>>>>>> This patch fixes globbing expansion in the backup include file list
>>>>>>> which got broken in c7e0d73e7cfd7be95db9d0a5f3392b8241813d5b.
>>>>>>> 
>>>>>>> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
>>>>>>> ---
>>>>>>>   config/backup/backup.pl | 8 ++++----
>>>>>>>   1 file changed, 4 insertions(+), 4 deletions(-)
>>>>>>> 
>>>>>>> diff --git a/config/backup/backup.pl b/config/backup/backup.pl
>>>>>>> index a2337cf23..6f9295e94 100644
>>>>>>> --- a/config/backup/backup.pl
>>>>>>> +++ b/config/backup/backup.pl
>>>>>>> @@ -19,6 +19,8 @@
>>>>>>> # #
>>>>>>> ############################################################################### 
>>>>>>>   +shopt -s nullglob
>>>>>>> +
>>>>>>>   NOW="$(date "+%Y-%m-%d-%H:%M")"
>>>>>>>     list_addons() {
>>>>>>> @@ -38,10 +40,8 @@ process_includes() {
>>>>>>>       for include in $@; do
>>>>>>>           local file
>>>>>>>           while read -r file; do
>>>>>>> -            for file in ${file}; do
>>>>>>> -                if [ -e "/${file}" ]; then
>>>>>>> -                    echo "${file}"
>>>>>>> -                fi
>>>>>>> +            for file in /${file}; do
>>>>>>> +                echo "${file}"
>>>>>>>               done
>>>>>>>           done < "${include}"
>>>>>>>       done | sort -u
>>>>>

Adolf Belka March 29, 2022, 3:23 p.m. UTC | #11

Hi All,

On 29/03/2022 16:22, Michael Tremer wrote:
> 
>> On 29 Mar 2022, at 15:22, Bernhard Bitsch <bbitsch@ipfire.org> wrote:
>>
>> Looks like your include.user file contains absolute paths.
That was not the problem but I have corrected that. I would have expected that when the main include and exclude files were made relative that the update script would also have checked for any entries in the include.user and exclude.user files but it looks like it didn't.
> 
> Or an empty line maybe?
That turned out to be the problem. There was an empty line after the two lines I had added. You only saw it if you scrolled the pointer down the file in the editor.

Removing that blank line removed the line only with a / and the backup then successfully ran and created a backup file containing all the directories expected.

The only thing I found is that there was the following tar error message

tar: Exiting with failure status due to previous errors

The only thing I found in the backup output was several lines such as

tar: /etc/squid/squid.conf.local: Cannot stat: No such file or directory
tar: /etc/squid/squid.conf.pre.local: Cannot stat: No such file or directory

There were seven lines in total like this and the three I checked were files that did not exist on my system. I presume that these are files that should be backed up if present but don't have to be present so that the error messages are nothing to worry about.
There were no error messages when doing the backup via the WUI.

The blank line was only on my vm testbed system but as I clone this for any testing activities I will have to check all of my existing vm's to fix this.

Glad we resolved this simply. Sorry for the anxiety.

Regards,

Adolf.

> 
>>
>> Am 29.03.2022 um 16:14 schrieb Adolf Belka:
>>> Hi All,
>>> This is what I get with the pushd/popd version in place on the same system
>>> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
>>> etc/group
>>> etc/hosts
>>> etc/hosts.allow
>>> etc/hosts.deny
>>> etc/httpd/server.crt
>>> etc/httpd/server.csr
>>> etc/httpd/server-ecdsa.crt
>>> etc/httpd/server-ecdsa.csr
>>> etc/httpd/server-ecdsa.key
>>> etc/httpd/server.key
>>> etc/ipsec.user.conf
>>> etc/ipsec.user.secrets
>>> etc/logrotate.d
>>> etc/passwd
>>> etc/shadow
>>> etc/ssh/sshd_config
>>> etc/ssh/ssh_host_ecdsa_key
>>> etc/ssh/ssh_host_ecdsa_key.pub
>>> etc/ssh/ssh_host_ed25519_key
>>> etc/ssh/ssh_host_ed25519_key.pub
>>> etc/ssh/ssh_host_rsa_key
>>> etc/ssh/ssh_host_rsa_key.pub
>>> /etc/sudoers
>>> etc/sysconfig/createfiles
>>> etc/sysconfig/firewall.local
>>> etc/sysconfig/lm_sensors
>>> etc/sysconfig/modules
>>> etc/sysconfig/ramdisk
>>> etc/sysconfig/rc
>>> etc/sysconfig/rc.local
>>> etc/unbound
>>> /home/ahb
>>> root/.bash_history
>>> var/ipfire/accounting/settings.conf
>>> var/ipfire/auth/users
>>> var/ipfire/backup/addons/backup
>>> var/ipfire/backup/exclude.user
>>> var/ipfire/backup/include.user
>>> var/ipfire/ca/cacert.pem
>>> var/ipfire/captive/agb.txt
>>> var/ipfire/captive/clients
>>> var/ipfire/captive/coupons
>>> var/ipfire/captive/logo.dat
>>> var/ipfire/captive/settings
>>> var/ipfire/captive/terms.txt
>>> var/ipfire/captive/voucher_out
>>> var/ipfire/certs/hostcert.pem
>>> var/ipfire/certs/hostkey.pem
>>> var/ipfire/certs/phoebevmipseccert.pem
>>> var/ipfire/connscheduler/connscheduler.conf
>>> var/ipfire/crls/cacrl.pem
>>> var/ipfire/cups/cups-browsed.conf
>>> var/ipfire/cups/subscriptions.conf
>>> var/ipfire/ddns/config
>>> var/ipfire/ddns/ddns.conf
>>> var/ipfire/ddns/settings
>>> var/ipfire/dhcp/advoptions
>>> var/ipfire/dhcp/advoptions-list
>>> var/ipfire/dhcpc/dhcpcd.conf
>>> var/ipfire/dhcp/dhcpd.conf
>>> var/ipfire/dhcp/dhcpd.conf.local
>>> var/ipfire/dhcp/enable_blue
>>> var/ipfire/dhcp/enable_green
>>> var/ipfire/dhcp/fixleases
>>> var/ipfire/dhcp/settings
>>> var/ipfire/dma/auth.conf
>>> var/ipfire/dma/dma.conf
>>> var/ipfire/dma/mail.conf
>>> var/ipfire/dns
>>> var/ipfire/dnsforward/config
>>> var/ipfire/dns/settings
>>> var/ipfire/ethernet/aliases
>>> var/ipfire/ethernet/settings
>>> var/ipfire/ethernet/wireless
>>> var/ipfire/extrahd/settings
>>> var/ipfire/firewall
>>> var/ipfire/firewall/config
>>> var/ipfire/firewall/settings
>>> var/ipfire/fwhosts
>>> var/ipfire/isdn/settings
>>> var/ipfire/logging/settings
>>> var/ipfire/mac/settings
>>> var/ipfire/main/firstsetup_ok
>>> var/ipfire/main/gpl_accepted
>>> var/ipfire/main/hostname.conf
>>> var/ipfire/main/hosts
>>> var/ipfire/main/manualpages
>>> var/ipfire/main/routing
>>> var/ipfire/main/security
>>> var/ipfire/main/send_profile
>>> var/ipfire/main/settings
>>> var/ipfire/modem/settings
>>> var/ipfire/optionsfw/settings
>>> var/ipfire/ovpn
>>> var/ipfire/ovpn/ccd.conf
>>> var/ipfire/ovpn/collectd.vpn
>>> var/ipfire/ovpn/enable
>>> var/ipfire/ovpn/server.conf
>>> var/ipfire/ovpn/settings
>>> var/ipfire/pakfire/settings
>>> var/ipfire/ppp
>>> var/ipfire/ppp/fake-resolv.conf
>>> var/ipfire/ppp/settings
>>> var/ipfire/private/cakey.pem
>>> var/ipfire/proxy
>>> var/ipfire/proxy/asnbl-helper.conf
>>> var/ipfire/proxy/cachemgr.conf
>>> var/ipfire/proxy/enable
>>> var/ipfire/proxy/settings
>>> var/ipfire/proxy/squid.conf
>>> var/ipfire/qos/bin
>>> var/ipfire/qos/classes
>>> var/ipfire/qos/level7config
>>> var/ipfire/qos/portconfig
>>> var/ipfire/qos/settings
>>> var/ipfire/qos/subclasses
>>> var/ipfire/qos/tosconfig
>>> var/ipfire/remote/enablessh
>>> var/ipfire/remote/settings
>>> var/ipfire/sensors/settings
>>> var/ipfire/suricata/oinkmaster.conf
>>> var/ipfire/suricata/oinkmaster-modify-sids.conf
>>> var/ipfire/suricata/oinkmaster-provider-includes.conf
>>> var/ipfire/suricata/providers-settings
>>> var/ipfire/suricata/settings
>>> var/ipfire/suricata/suricata-default-rules.yaml
>>> var/ipfire/suricata/suricata-dns-servers.yaml
>>> var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
>>> var/ipfire/suricata/suricata-homenet.yaml
>>> var/ipfire/suricata/suricata-http-ports.yaml
>>> var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml
>>> var/ipfire/suricata/suricata-used-providers.yaml
>>> var/ipfire/time/
>>> var/ipfire/time/counter.conf
>>> var/ipfire/time/enable
>>> var/ipfire/time/settime.conf
>>> var/ipfire/time/settings
>>> var/ipfire/upnp/settings
>>> var/ipfire/urlfilter
>>> var/ipfire/urlfilter/settings
>>> var/ipfire/urlfilter/squidGuard.conf
>>> var/ipfire/vpn
>>> var/ipfire/vpn/config
>>> var/ipfire/vpn/ipsec.conf
>>> var/ipfire/vpn/settings
>>> var/ipfire/wakeonlan/clients.conf
>>> var/ipfire/wio/wio.conf
>>> var/ipfire/wireless/config
>>> var/ipfire/wireless/settings
>>> var/lib/suricata
>>> var/log/rrd/collectd
>>> var/log/rrd/hddshutdown-md127.rrd
>>> var/log/rrd/hddshutdown-sda.rrd
>>> var/log/rrd/hddshutdown-sdb.rrd
>>> var/log/rrd/hddtemp-md127.rrd
>>> var/log/rrd/hddtemp-sda.rrd
>>> var/log/rrd/hddtemp-sdb.rrd
>>> var/log/rrd/wio
>>> var/log/vnstat
>>> var/tmp/idsrules-emerging.tar.gz
>>> var/tmp/idsrules-sslbl_blacklist.rules
>>> The following are in the previous list but not in this one:-
>>> /
>>> /etc/conntrackd/conntrackd.conf
>>> /etc/ipsec.user-post.conf
>>> /root/.gitconfig
>>> /root/.ssh
>>> Regards,
>>> Adolf.
>>> On 29/03/2022 16:01, Adolf Belka wrote:
>>>> Hi All,
>>>>
>>>> On 29/03/2022 15:36, Bernhard Bitsch wrote:
>>>>> Hi,
>>>>>
>>>>> Am 29.03.2022 um 15:11 schrieb Michael Tremer:
>>>>>> Hello,
>>>>>>
>>>>>> Could you please send the file listing to find out what is being included what shouldn’t?
>>>> Unfortunately, as I stopped the backup continuing once it had reached 1.2GB, the file created was not able to be opened. Probably stopping the backup corrupted it in some way.
>>>>>>
>>>>>
>>>>> Just for easy quick handling, you can use
>>>>> '/var/ipfire/backup/bin/backup.pl list' to show the files included.
>>>> I gave this a go and here is the output from the command.
>>>>
>>>> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
>>>> /
>>>> /etc/conntrackd/conntrackd.conf
>>>> /etc/group
>>>> /etc/hosts
>>>> /etc/hosts.allow
>>>> /etc/hosts.deny
>>>> /etc/httpd/server.crt
>>>> /etc/httpd/server.csr
>>>> /etc/httpd/server-ecdsa.crt
>>>> /etc/httpd/server-ecdsa.csr
>>>> /etc/httpd/server-ecdsa.key
>>>> /etc/httpd/server.key
>>>> /etc/ipsec.user.conf
>>>> /etc/ipsec.user-post.conf
>>>> /etc/ipsec.user.secrets
>>>> /etc/logrotate.d
>>>> /etc/passwd
>>>> /etc/shadow
>>>> /etc/squid/squid.conf.local
>>>> /etc/squid/squid.conf.pre.local
>>>> /etc/ssh/sshd_config
>>>> /etc/ssh/ssh_host_ecdsa_key
>>>> /etc/ssh/ssh_host_ecdsa_key.pub
>>>> /etc/ssh/ssh_host_ed25519_key
>>>> /etc/ssh/ssh_host_ed25519_key.pub
>>>> /etc/ssh/ssh_host_rsa_key
>>>> /etc/ssh/ssh_host_rsa_key.pub
>>>> //etc/sudoers
>>>> /etc/sysconfig/createfiles
>>>> /etc/sysconfig/firewall.local
>>>> /etc/sysconfig/lm_sensors
>>>> /etc/sysconfig/modules
>>>> /etc/sysconfig/ramdisk
>>>> /etc/sysconfig/rc
>>>> /etc/sysconfig/rc.local
>>>> /etc/unbound
>>>> //home/ahb
>>>> /root/.bash_history
>>>> /root/.gitconfig
>>>> /root/.ssh
>>>> /var/ipfire/accounting/settings.conf
>>>> /var/ipfire/auth/users
>>>> /var/ipfire/backup/addons/backup
>>>> /var/ipfire/backup/exclude.user
>>>> /var/ipfire/backup/include.user
>>>> /var/ipfire/ca/cacert.pem
>>>> /var/ipfire/captive/agb.txt
>>>> /var/ipfire/captive/clients
>>>> /var/ipfire/captive/coupons
>>>> /var/ipfire/captive/logo.dat
>>>> /var/ipfire/captive/settings
>>>> /var/ipfire/captive/terms.txt
>>>> /var/ipfire/captive/voucher_out
>>>> /var/ipfire/certs/hostcert.pem
>>>> /var/ipfire/certs/hostkey.pem
>>>> /var/ipfire/certs/phoebevmipseccert.pem
>>>> /var/ipfire/connscheduler/connscheduler.conf
>>>> /var/ipfire/crls/cacrl.pem
>>>> /var/ipfire/cups/cups-browsed.conf
>>>> /var/ipfire/cups/subscriptions.conf
>>>> /var/ipfire/ddns/config
>>>> /var/ipfire/ddns/ddns.conf
>>>> /var/ipfire/ddns/settings
>>>> /var/ipfire/dhcp/advoptions
>>>> /var/ipfire/dhcp/advoptions-list
>>>> /var/ipfire/dhcpc/dhcpcd.conf
>>>> /var/ipfire/dhcp/dhcpd.conf
>>>> /var/ipfire/dhcp/dhcpd.conf.local
>>>> /var/ipfire/dhcp/enable_blue
>>>> /var/ipfire/dhcp/enable_green
>>>> /var/ipfire/dhcp/fixleases
>>>> /var/ipfire/dhcp/settings
>>>> /var/ipfire/dma/auth.conf
>>>> /var/ipfire/dma/dma.conf
>>>> /var/ipfire/dma/mail.conf
>>>> /var/ipfire/dns
>>>> /var/ipfire/dnsforward/config
>>>> /var/ipfire/dns/settings
>>>> /var/ipfire/ethernet/aliases
>>>> /var/ipfire/ethernet/settings
>>>> /var/ipfire/ethernet/wireless
>>>> /var/ipfire/extrahd/settings
>>>> /var/ipfire/firewall
>>>> /var/ipfire/firewall/config
>>>> /var/ipfire/firewall/settings
>>>> /var/ipfire/fwhosts
>>>> /var/ipfire/isdn/settings
>>>> /var/ipfire/logging/settings
>>>> /var/ipfire/mac/settings
>>>> /var/ipfire/main/firstsetup_ok
>>>> /var/ipfire/main/gpl_accepted
>>>> /var/ipfire/main/hostname.conf
>>>> /var/ipfire/main/hosts
>>>> /var/ipfire/main/manualpages
>>>> /var/ipfire/main/routing
>>>> /var/ipfire/main/security
>>>> /var/ipfire/main/send_profile
>>>> /var/ipfire/main/settings
>>>> /var/ipfire/modem/settings
>>>> /var/ipfire/optionsfw/settings
>>>> /var/ipfire/ovpn
>>>> /var/ipfire/ovpn/ccd.conf
>>>> /var/ipfire/ovpn/collectd.vpn
>>>> /var/ipfire/ovpn/enable
>>>> /var/ipfire/ovpn/server.conf
>>>> /var/ipfire/ovpn/settings
>>>> /var/ipfire/pakfire/settings
>>>> /var/ipfire/ppp
>>>> /var/ipfire/ppp/fake-resolv.conf
>>>> /var/ipfire/ppp/settings
>>>> /var/ipfire/private/cakey.pem
>>>> /var/ipfire/proxy
>>>> /var/ipfire/proxy/asnbl-helper.conf
>>>> /var/ipfire/proxy/cachemgr.conf
>>>> /var/ipfire/proxy/enable
>>>> /var/ipfire/proxy/settings
>>>> /var/ipfire/proxy/squid.conf
>>>> /var/ipfire/qos/bin
>>>> /var/ipfire/qos/bin/qos.sh
>>>> /var/ipfire/qos/classes
>>>> /var/ipfire/qos/level7config
>>>> /var/ipfire/qos/portconfig
>>>> /var/ipfire/qos/settings
>>>> /var/ipfire/qos/subclasses
>>>> /var/ipfire/qos/tosconfig
>>>> /var/ipfire/remote/enablessh
>>>> /var/ipfire/remote/settings
>>>> /var/ipfire/sensors/settings
>>>> /var/ipfire/suricata/oinkmaster.conf
>>>> /var/ipfire/suricata/oinkmaster-modify-sids.conf
>>>> /var/ipfire/suricata/oinkmaster-provider-includes.conf
>>>> /var/ipfire/suricata/providers-settings
>>>> /var/ipfire/suricata/settings
>>>> /var/ipfire/suricata/suricata-default-rules.yaml
>>>> /var/ipfire/suricata/suricata-dns-servers.yaml
>>>> /var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
>>>> /var/ipfire/suricata/suricata-homenet.yaml
>>>> /var/ipfire/suricata/suricata-http-ports.yaml
>>>> /var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml
>>>> /var/ipfire/suricata/suricata-used-providers.yaml
>>>> /var/ipfire/time/
>>>> /var/ipfire/time/counter.conf
>>>> /var/ipfire/time/enable
>>>> /var/ipfire/time/settime.conf
>>>> /var/ipfire/time/settings
>>>> /var/ipfire/upnp/settings
>>>> /var/ipfire/urlfilter
>>>> /var/ipfire/urlfilter/settings
>>>> /var/ipfire/urlfilter/squidGuard.conf
>>>> /var/ipfire/vpn
>>>> /var/ipfire/vpn/config
>>>> /var/ipfire/vpn/ipsec.conf
>>>> /var/ipfire/vpn/settings
>>>> /var/ipfire/wakeonlan/clients.conf
>>>> /var/ipfire/wio/wio.conf
>>>> /var/ipfire/wireless/config
>>>> /var/ipfire/wireless/settings
>>>> /var/lib/suricata
>>>> /var/log/rrd/collectd
>>>> /var/log/rrd/hddshutdown-md127.rrd
>>>> /var/log/rrd/hddshutdown-sda.rrd
>>>> /var/log/rrd/hddshutdown-sdb.rrd
>>>> /var/log/rrd/hddtemp-md127.rrd
>>>> /var/log/rrd/hddtemp-sda.rrd
>>>> /var/log/rrd/hddtemp-sdb.rrd
>>>> /var/log/rrd/wio
>>>> /var/log/vnstat
>>>> /var/tmp/idsrules-emerging.tar.gz
>>>> /var/tmp/idsrules-sslbl_blacklist.rules
>>>>
>>>> Regards,
>>>> Adolf.
>>>>>
>>>>> Bernhard
>>>>>> -Michael
>>>>>>
>>>>>>> On 29 Mar 2022, at 14:10, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>>>>>>
>>>>>>> Hi Michael,
>>>>>>>
>>>>>>> Just tried this patch out on my vm testbed system and it still doesn't work for me. The backup file had got to 1.3GB when I deleted the backup file as it was still growing. The normal correct backup file on that vm machine is around 7MB
>>>>>>>
>>>>>>> The above was the case for both running it from the WUI or from the command line from my unprivileged user using sudo backupctrl exclude
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Adolf.
>>>>>>>
>>>>>>>
>>>>>>> On 29/03/2022 14:27, Michael Tremer wrote:
>>>>>>>> This patch fixes globbing expansion in the backup include file list
>>>>>>>> which got broken in c7e0d73e7cfd7be95db9d0a5f3392b8241813d5b.
>>>>>>>>
>>>>>>>> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
>>>>>>>> ---
>>>>>>>>    config/backup/backup.pl | 8 ++++----
>>>>>>>>    1 file changed, 4 insertions(+), 4 deletions(-)
>>>>>>>>
>>>>>>>> diff --git a/config/backup/backup.pl b/config/backup/backup.pl
>>>>>>>> index a2337cf23..6f9295e94 100644
>>>>>>>> --- a/config/backup/backup.pl
>>>>>>>> +++ b/config/backup/backup.pl
>>>>>>>> @@ -19,6 +19,8 @@
>>>>>>>> # #
>>>>>>>> ###############################################################################
>>>>>>>>    +shopt -s nullglob
>>>>>>>> +
>>>>>>>>    NOW="$(date "+%Y-%m-%d-%H:%M")"
>>>>>>>>      list_addons() {
>>>>>>>> @@ -38,10 +40,8 @@ process_includes() {
>>>>>>>>        for include in $@; do
>>>>>>>>            local file
>>>>>>>>            while read -r file; do
>>>>>>>> -            for file in ${file}; do
>>>>>>>> -                if [ -e "/${file}" ]; then
>>>>>>>> -                    echo "${file}"
>>>>>>>> -                fi
>>>>>>>> +            for file in /${file}; do
>>>>>>>> +                echo "${file}"
>>>>>>>>                done
>>>>>>>>            done < "${include}"
>>>>>>>>        done | sort -u
>>>>>>
>

Bernhard Bitsch March 29, 2022, 6:36 p.m. UTC | #12

Hi all,

Am 29.03.2022 um 17:23 schrieb Adolf Belka:
> Hi All,
> 
> On 29/03/2022 16:22, Michael Tremer wrote:
>>
>>> On 29 Mar 2022, at 15:22, Bernhard Bitsch <bbitsch@ipfire.org> wrote:
>>>
>>> Looks like your include.user file contains absolute paths.
> That was not the problem but I have corrected that. I would have 
> expected that when the main include and exclude files were made relative 
> that the update script would also have checked for any entries in the 
> include.user and exclude.user files but it looks like it didn't.th
According to the source they are chosen.
>>
>> Or an empty line maybe?
> That turned out to be the problem. There was an empty line after the two 
> lines I had added. You only saw it if you scrolled the pointer down the 
> file in the editor.
Interesting. A blank line should produce nothing.
> 
> Removing that blank line removed the line only with a / and the backup 
> then successfully ran and created a backup file containing all the 
> directories expected.
> 
> The only thing I found is that there was the following tar error message
> 
> tar: Exiting with failure status due to previous errors
> 
> The only thing I found in the backup output was several lines such as
> 
> tar: /etc/squid/squid.conf.local: Cannot stat: No such file or directory
> tar: /etc/squid/squid.conf.pre.local: Cannot stat: No such file or 
> directory
> 
> There were seven lines in total like this and the three I checked were 
> files that did not exist on my system. I presume that these are files 
> that should be backed up if present but don't have to be present so that 
> the error messages are nothing to worry about.
> There were no error messages when doing the backup via the WUI.
> 

Good find. ;)
Michael's patch removes the test of existence, globbing produces 
existent file names only. But I suppose there are several full names of 
files, not existing in each implementation.
Conclusion: Repair the file globbing by the shopt and do the existence 
check. So all existent files are chosen, that are listed in the include 
files.

Regards,
Bernhard

> The blank line was only on my vm testbed system but as I clone this for 
> any testing activities I will have to check all of my existing vm's to 
> fix this.
> 
> Glad we resolved this simply. Sorry for the anxiety.
> 
> Regards,
> 
> Adolf.
> 
>>
>>>
>>> Am 29.03.2022 um 16:14 schrieb Adolf Belka:
>>>> Hi All,
>>>> This is what I get with the pushd/popd version in place on the same 
>>>> system
>>>> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
>>>> etc/group
>>>> etc/hosts
>>>> etc/hosts.allow
>>>> etc/hosts.deny
>>>> etc/httpd/server.crt
>>>> etc/httpd/server.csr
>>>> etc/httpd/server-ecdsa.crt
>>>> etc/httpd/server-ecdsa.csr
>>>> etc/httpd/server-ecdsa.key
>>>> etc/httpd/server.key
>>>> etc/ipsec.user.conf
>>>> etc/ipsec.user.secrets
>>>> etc/logrotate.d
>>>> etc/passwd
>>>> etc/shadow
>>>> etc/ssh/sshd_config
>>>> etc/ssh/ssh_host_ecdsa_key
>>>> etc/ssh/ssh_host_ecdsa_key.pub
>>>> etc/ssh/ssh_host_ed25519_key
>>>> etc/ssh/ssh_host_ed25519_key.pub
>>>> etc/ssh/ssh_host_rsa_key
>>>> etc/ssh/ssh_host_rsa_key.pub
>>>> /etc/sudoers
>>>> etc/sysconfig/createfiles
>>>> etc/sysconfig/firewall.local
>>>> etc/sysconfig/lm_sensors
>>>> etc/sysconfig/modules
>>>> etc/sysconfig/ramdisk
>>>> etc/sysconfig/rc
>>>> etc/sysconfig/rc.local
>>>> etc/unbound
>>>> /home/ahb
>>>> root/.bash_history
>>>> var/ipfire/accounting/settings.conf
>>>> var/ipfire/auth/users
>>>> var/ipfire/backup/addons/backup
>>>> var/ipfire/backup/exclude.user
>>>> var/ipfire/backup/include.user
>>>> var/ipfire/ca/cacert.pem
>>>> var/ipfire/captive/agb.txt
>>>> var/ipfire/captive/clients
>>>> var/ipfire/captive/coupons
>>>> var/ipfire/captive/logo.dat
>>>> var/ipfire/captive/settings
>>>> var/ipfire/captive/terms.txt
>>>> var/ipfire/captive/voucher_out
>>>> var/ipfire/certs/hostcert.pem
>>>> var/ipfire/certs/hostkey.pem
>>>> var/ipfire/certs/phoebevmipseccert.pem
>>>> var/ipfire/connscheduler/connscheduler.conf
>>>> var/ipfire/crls/cacrl.pem
>>>> var/ipfire/cups/cups-browsed.conf
>>>> var/ipfire/cups/subscriptions.conf
>>>> var/ipfire/ddns/config
>>>> var/ipfire/ddns/ddns.conf
>>>> var/ipfire/ddns/settings
>>>> var/ipfire/dhcp/advoptions
>>>> var/ipfire/dhcp/advoptions-list
>>>> var/ipfire/dhcpc/dhcpcd.conf
>>>> var/ipfire/dhcp/dhcpd.conf
>>>> var/ipfire/dhcp/dhcpd.conf.local
>>>> var/ipfire/dhcp/enable_blue
>>>> var/ipfire/dhcp/enable_green
>>>> var/ipfire/dhcp/fixleases
>>>> var/ipfire/dhcp/settings
>>>> var/ipfire/dma/auth.conf
>>>> var/ipfire/dma/dma.conf
>>>> var/ipfire/dma/mail.conf
>>>> var/ipfire/dns
>>>> var/ipfire/dnsforward/config
>>>> var/ipfire/dns/settings
>>>> var/ipfire/ethernet/aliases
>>>> var/ipfire/ethernet/settings
>>>> var/ipfire/ethernet/wireless
>>>> var/ipfire/extrahd/settings
>>>> var/ipfire/firewall
>>>> var/ipfire/firewall/config
>>>> var/ipfire/firewall/settings
>>>> var/ipfire/fwhosts
>>>> var/ipfire/isdn/settings
>>>> var/ipfire/logging/settings
>>>> var/ipfire/mac/settings
>>>> var/ipfire/main/firstsetup_ok
>>>> var/ipfire/main/gpl_accepted
>>>> var/ipfire/main/hostname.conf
>>>> var/ipfire/main/hosts
>>>> var/ipfire/main/manualpages
>>>> var/ipfire/main/routing
>>>> var/ipfire/main/security
>>>> var/ipfire/main/send_profile
>>>> var/ipfire/main/settings
>>>> var/ipfire/modem/settings
>>>> var/ipfire/optionsfw/settings
>>>> var/ipfire/ovpn
>>>> var/ipfire/ovpn/ccd.conf
>>>> var/ipfire/ovpn/collectd.vpn
>>>> var/ipfire/ovpn/enable
>>>> var/ipfire/ovpn/server.conf
>>>> var/ipfire/ovpn/settings
>>>> var/ipfire/pakfire/settings
>>>> var/ipfire/ppp
>>>> var/ipfire/ppp/fake-resolv.conf
>>>> var/ipfire/ppp/settings
>>>> var/ipfire/private/cakey.pem
>>>> var/ipfire/proxy
>>>> var/ipfire/proxy/asnbl-helper.conf
>>>> var/ipfire/proxy/cachemgr.conf
>>>> var/ipfire/proxy/enable
>>>> var/ipfire/proxy/settings
>>>> var/ipfire/proxy/squid.conf
>>>> var/ipfire/qos/bin
>>>> var/ipfire/qos/classes
>>>> var/ipfire/qos/level7config
>>>> var/ipfire/qos/portconfig
>>>> var/ipfire/qos/settings
>>>> var/ipfire/qos/subclasses
>>>> var/ipfire/qos/tosconfig
>>>> var/ipfire/remote/enablessh
>>>> var/ipfire/remote/settings
>>>> var/ipfire/sensors/settings
>>>> var/ipfire/suricata/oinkmaster.conf
>>>> var/ipfire/suricata/oinkmaster-modify-sids.conf
>>>> var/ipfire/suricata/oinkmaster-provider-includes.conf
>>>> var/ipfire/suricata/providers-settings
>>>> var/ipfire/suricata/settings
>>>> var/ipfire/suricata/suricata-default-rules.yaml
>>>> var/ipfire/suricata/suricata-dns-servers.yaml
>>>> var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
>>>> var/ipfire/suricata/suricata-homenet.yaml
>>>> var/ipfire/suricata/suricata-http-ports.yaml
>>>> var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml
>>>> var/ipfire/suricata/suricata-used-providers.yaml
>>>> var/ipfire/time/
>>>> var/ipfire/time/counter.conf
>>>> var/ipfire/time/enable
>>>> var/ipfire/time/settime.conf
>>>> var/ipfire/time/settings
>>>> var/ipfire/upnp/settings
>>>> var/ipfire/urlfilter
>>>> var/ipfire/urlfilter/settings
>>>> var/ipfire/urlfilter/squidGuard.conf
>>>> var/ipfire/vpn
>>>> var/ipfire/vpn/config
>>>> var/ipfire/vpn/ipsec.conf
>>>> var/ipfire/vpn/settings
>>>> var/ipfire/wakeonlan/clients.conf
>>>> var/ipfire/wio/wio.conf
>>>> var/ipfire/wireless/config
>>>> var/ipfire/wireless/settings
>>>> var/lib/suricata
>>>> var/log/rrd/collectd
>>>> var/log/rrd/hddshutdown-md127.rrd
>>>> var/log/rrd/hddshutdown-sda.rrd
>>>> var/log/rrd/hddshutdown-sdb.rrd
>>>> var/log/rrd/hddtemp-md127.rrd
>>>> var/log/rrd/hddtemp-sda.rrd
>>>> var/log/rrd/hddtemp-sdb.rrd
>>>> var/log/rrd/wio
>>>> var/log/vnstat
>>>> var/tmp/idsrules-emerging.tar.gz
>>>> var/tmp/idsrules-sslbl_blacklist.rules
>>>> The following are in the previous list but not in this one:-
>>>> /
>>>> /etc/conntrackd/conntrackd.conf
>>>> /etc/ipsec.user-post.conf
>>>> /root/.gitconfig
>>>> /root/.ssh
>>>> Regards,
>>>> Adolf.
>>>> On 29/03/2022 16:01, Adolf Belka wrote:
>>>>> Hi All,
>>>>>
>>>>> On 29/03/2022 15:36, Bernhard Bitsch wrote:
>>>>>> Hi,
>>>>>>
>>>>>> Am 29.03.2022 um 15:11 schrieb Michael Tremer:
>>>>>>> Hello,
>>>>>>>
>>>>>>> Could you please send the file listing to find out what is being 
>>>>>>> included what shouldn’t?
>>>>> Unfortunately, as I stopped the backup continuing once it had 
>>>>> reached 1.2GB, the file created was not able to be opened. Probably 
>>>>> stopping the backup corrupted it in some way.
>>>>>>>
>>>>>>
>>>>>> Just for easy quick handling, you can use
>>>>>> '/var/ipfire/backup/bin/backup.pl list' to show the files included.
>>>>> I gave this a go and here is the output from the command.
>>>>>
>>>>> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
>>>>> /
>>>>> /etc/conntrackd/conntrackd.conf
>>>>> /etc/group
>>>>> /etc/hosts
>>>>> /etc/hosts.allow
>>>>> /etc/hosts.deny
>>>>> /etc/httpd/server.crt
>>>>> /etc/httpd/server.csr
>>>>> /etc/httpd/server-ecdsa.crt
>>>>> /etc/httpd/server-ecdsa.csr
>>>>> /etc/httpd/server-ecdsa.key
>>>>> /etc/httpd/server.key
>>>>> /etc/ipsec.user.conf
>>>>> /etc/ipsec.user-post.conf
>>>>> /etc/ipsec.user.secrets
>>>>> /etc/logrotate.d
>>>>> /etc/passwd
>>>>> /etc/shadow
>>>>> /etc/squid/squid.conf.local
>>>>> /etc/squid/squid.conf.pre.local
>>>>> /etc/ssh/sshd_config
>>>>> /etc/ssh/ssh_host_ecdsa_key
>>>>> /etc/ssh/ssh_host_ecdsa_key.pub
>>>>> /etc/ssh/ssh_host_ed25519_key
>>>>> /etc/ssh/ssh_host_ed25519_key.pub
>>>>> /etc/ssh/ssh_host_rsa_key
>>>>> /etc/ssh/ssh_host_rsa_key.pub
>>>>> //etc/sudoers
>>>>> /etc/sysconfig/createfiles
>>>>> /etc/sysconfig/firewall.local
>>>>> /etc/sysconfig/lm_sensors
>>>>> /etc/sysconfig/modules
>>>>> /etc/sysconfig/ramdisk
>>>>> /etc/sysconfig/rc
>>>>> /etc/sysconfig/rc.local
>>>>> /etc/unbound
>>>>> //home/ahb
>>>>> /root/.bash_history
>>>>> /root/.gitconfig
>>>>> /root/.ssh
>>>>> /var/ipfire/accounting/settings.conf
>>>>> /var/ipfire/auth/users
>>>>> /var/ipfire/backup/addons/backup
>>>>> /var/ipfire/backup/exclude.user
>>>>> /var/ipfire/backup/include.user
>>>>> /var/ipfire/ca/cacert.pem
>>>>> /var/ipfire/captive/agb.txt
>>>>> /var/ipfire/captive/clients
>>>>> /var/ipfire/captive/coupons
>>>>> /var/ipfire/captive/logo.dat
>>>>> /var/ipfire/captive/settings
>>>>> /var/ipfire/captive/terms.txt
>>>>> /var/ipfire/captive/voucher_out
>>>>> /var/ipfire/certs/hostcert.pem
>>>>> /var/ipfire/certs/hostkey.pem
>>>>> /var/ipfire/certs/phoebevmipseccert.pem
>>>>> /var/ipfire/connscheduler/connscheduler.conf
>>>>> /var/ipfire/crls/cacrl.pem
>>>>> /var/ipfire/cups/cups-browsed.conf
>>>>> /var/ipfire/cups/subscriptions.conf
>>>>> /var/ipfire/ddns/config
>>>>> /var/ipfire/ddns/ddns.conf
>>>>> /var/ipfire/ddns/settings
>>>>> /var/ipfire/dhcp/advoptions
>>>>> /var/ipfire/dhcp/advoptions-list
>>>>> /var/ipfire/dhcpc/dhcpcd.conf
>>>>> /var/ipfire/dhcp/dhcpd.conf
>>>>> /var/ipfire/dhcp/dhcpd.conf.local
>>>>> /var/ipfire/dhcp/enable_blue
>>>>> /var/ipfire/dhcp/enable_green
>>>>> /var/ipfire/dhcp/fixleases
>>>>> /var/ipfire/dhcp/settings
>>>>> /var/ipfire/dma/auth.conf
>>>>> /var/ipfire/dma/dma.conf
>>>>> /var/ipfire/dma/mail.conf
>>>>> /var/ipfire/dns
>>>>> /var/ipfire/dnsforward/config
>>>>> /var/ipfire/dns/settings
>>>>> /var/ipfire/ethernet/aliases
>>>>> /var/ipfire/ethernet/settings
>>>>> /var/ipfire/ethernet/wireless
>>>>> /var/ipfire/extrahd/settings
>>>>> /var/ipfire/firewall
>>>>> /var/ipfire/firewall/config
>>>>> /var/ipfire/firewall/settings
>>>>> /var/ipfire/fwhosts
>>>>> /var/ipfire/isdn/settings
>>>>> /var/ipfire/logging/settings
>>>>> /var/ipfire/mac/settings
>>>>> /var/ipfire/main/firstsetup_ok
>>>>> /var/ipfire/main/gpl_accepted
>>>>> /var/ipfire/main/hostname.conf
>>>>> /var/ipfire/main/hosts
>>>>> /var/ipfire/main/manualpages
>>>>> /var/ipfire/main/routing
>>>>> /var/ipfire/main/security
>>>>> /var/ipfire/main/send_profile
>>>>> /var/ipfire/main/settings
>>>>> /var/ipfire/modem/settings
>>>>> /var/ipfire/optionsfw/settings
>>>>> /var/ipfire/ovpn
>>>>> /var/ipfire/ovpn/ccd.conf
>>>>> /var/ipfire/ovpn/collectd.vpn
>>>>> /var/ipfire/ovpn/enable
>>>>> /var/ipfire/ovpn/server.conf
>>>>> /var/ipfire/ovpn/settings
>>>>> /var/ipfire/pakfire/settings
>>>>> /var/ipfire/ppp
>>>>> /var/ipfire/ppp/fake-resolv.conf
>>>>> /var/ipfire/ppp/settings
>>>>> /var/ipfire/private/cakey.pem
>>>>> /var/ipfire/proxy
>>>>> /var/ipfire/proxy/asnbl-helper.conf
>>>>> /var/ipfire/proxy/cachemgr.conf
>>>>> /var/ipfire/proxy/enable
>>>>> /var/ipfire/proxy/settings
>>>>> /var/ipfire/proxy/squid.conf
>>>>> /var/ipfire/qos/bin
>>>>> /var/ipfire/qos/bin/qos.sh
>>>>> /var/ipfire/qos/classes
>>>>> /var/ipfire/qos/level7config
>>>>> /var/ipfire/qos/portconfig
>>>>> /var/ipfire/qos/settings
>>>>> /var/ipfire/qos/subclasses
>>>>> /var/ipfire/qos/tosconfig
>>>>> /var/ipfire/remote/enablessh
>>>>> /var/ipfire/remote/settings
>>>>> /var/ipfire/sensors/settings
>>>>> /var/ipfire/suricata/oinkmaster.conf
>>>>> /var/ipfire/suricata/oinkmaster-modify-sids.conf
>>>>> /var/ipfire/suricata/oinkmaster-provider-includes.conf
>>>>> /var/ipfire/suricata/providers-settings
>>>>> /var/ipfire/suricata/settings
>>>>> /var/ipfire/suricata/suricata-default-rules.yaml
>>>>> /var/ipfire/suricata/suricata-dns-servers.yaml
>>>>> /var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
>>>>> /var/ipfire/suricata/suricata-homenet.yaml
>>>>> /var/ipfire/suricata/suricata-http-ports.yaml
>>>>> /var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml
>>>>> /var/ipfire/suricata/suricata-used-providers.yaml
>>>>> /var/ipfire/time/
>>>>> /var/ipfire/time/counter.conf
>>>>> /var/ipfire/time/enable
>>>>> /var/ipfire/time/settime.conf
>>>>> /var/ipfire/time/settings
>>>>> /var/ipfire/upnp/settings
>>>>> /var/ipfire/urlfilter
>>>>> /var/ipfire/urlfilter/settings
>>>>> /var/ipfire/urlfilter/squidGuard.conf
>>>>> /var/ipfire/vpn
>>>>> /var/ipfire/vpn/config
>>>>> /var/ipfire/vpn/ipsec.conf
>>>>> /var/ipfire/vpn/settings
>>>>> /var/ipfire/wakeonlan/clients.conf
>>>>> /var/ipfire/wio/wio.conf
>>>>> /var/ipfire/wireless/config
>>>>> /var/ipfire/wireless/settings
>>>>> /var/lib/suricata
>>>>> /var/log/rrd/collectd
>>>>> /var/log/rrd/hddshutdown-md127.rrd
>>>>> /var/log/rrd/hddshutdown-sda.rrd
>>>>> /var/log/rrd/hddshutdown-sdb.rrd
>>>>> /var/log/rrd/hddtemp-md127.rrd
>>>>> /var/log/rrd/hddtemp-sda.rrd
>>>>> /var/log/rrd/hddtemp-sdb.rrd
>>>>> /var/log/rrd/wio
>>>>> /var/log/vnstat
>>>>> /var/tmp/idsrules-emerging.tar.gz
>>>>> /var/tmp/idsrules-sslbl_blacklist.rules
>>>>>
>>>>> Regards,
>>>>> Adolf.
>>>>>>
>>>>>> Bernhard
>>>>>>> -Michael
>>>>>>>
>>>>>>>> On 29 Mar 2022, at 14:10, Adolf Belka <adolf.belka@ipfire.org> 
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi Michael,
>>>>>>>>
>>>>>>>> Just tried this patch out on my vm testbed system and it still 
>>>>>>>> doesn't work for me. The backup file had got to 1.3GB when I 
>>>>>>>> deleted the backup file as it was still growing. The normal 
>>>>>>>> correct backup file on that vm machine is around 7MB
>>>>>>>>
>>>>>>>> The above was the case for both running it from the WUI or from 
>>>>>>>> the command line from my unprivileged user using sudo backupctrl 
>>>>>>>> exclude
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> Adolf.
>>>>>>>>
>>>>>>>>
>>>>>>>> On 29/03/2022 14:27, Michael Tremer wrote:
>>>>>>>>> This patch fixes globbing expansion in the backup include file 
>>>>>>>>> list
>>>>>>>>> which got broken in c7e0d73e7cfd7be95db9d0a5f3392b8241813d5b.
>>>>>>>>>
>>>>>>>>> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
>>>>>>>>> ---
>>>>>>>>>    config/backup/backup.pl | 8 ++++----
>>>>>>>>>    1 file changed, 4 insertions(+), 4 deletions(-)
>>>>>>>>>
>>>>>>>>> diff --git a/config/backup/backup.pl b/config/backup/backup.pl
>>>>>>>>> index a2337cf23..6f9295e94 100644
>>>>>>>>> --- a/config/backup/backup.pl
>>>>>>>>> +++ b/config/backup/backup.pl
>>>>>>>>> @@ -19,6 +19,8 @@
>>>>>>>>> # #
>>>>>>>>> ############################################################################### 
>>>>>>>>>
>>>>>>>>>    +shopt -s nullglob
>>>>>>>>> +
>>>>>>>>>    NOW="$(date "+%Y-%m-%d-%H:%M")"
>>>>>>>>>      list_addons() {
>>>>>>>>> @@ -38,10 +40,8 @@ process_includes() {
>>>>>>>>>        for include in $@; do
>>>>>>>>>            local file
>>>>>>>>>            while read -r file; do
>>>>>>>>> -            for file in ${file}; do
>>>>>>>>> -                if [ -e "/${file}" ]; then
>>>>>>>>> -                    echo "${file}"
>>>>>>>>> -                fi
>>>>>>>>> +            for file in /${file}; do
>>>>>>>>> +                echo "${file}"
>>>>>>>>>                done
>>>>>>>>>            done < "${include}"
>>>>>>>>>        done | sort -u
>>>>>>>
>>

Bernhard Bitsch March 29, 2022, 7:21 p.m. UTC | #13

Just checked the blank line problem.
Pattern is empty , file name list is '/' --> no names produced ( with 
existence check )
Pattern is empty --> produced file name '/' ( without -e check )

Am 29.03.2022 um 20:36 schrieb Bernhard Bitsch:
> Hi all,
> 
> Am 29.03.2022 um 17:23 schrieb Adolf Belka:
>> Hi All,
>>
>> On 29/03/2022 16:22, Michael Tremer wrote:
>>>
>>>> On 29 Mar 2022, at 15:22, Bernhard Bitsch <bbitsch@ipfire.org> wrote:
>>>>
>>>> Looks like your include.user file contains absolute paths.
>> That was not the problem but I have corrected that. I would have 
>> expected that when the main include and exclude files were made 
>> relative that the update script would also have checked for any 
>> entries in the include.user and exclude.user files but it looks like 
>> it didn't.th
> According to the source they are chosen.
>>>
>>> Or an empty line maybe?
>> That turned out to be the problem. There was an empty line after the 
>> two lines I had added. You only saw it if you scrolled the pointer 
>> down the file in the editor.
> Interesting. A blank line should produce nothing.
>>
>> Removing that blank line removed the line only with a / and the backup 
>> then successfully ran and created a backup file containing all the 
>> directories expected.
>>
>> The only thing I found is that there was the following tar error message
>>
>> tar: Exiting with failure status due to previous errors
>>
>> The only thing I found in the backup output was several lines such as
>>
>> tar: /etc/squid/squid.conf.local: Cannot stat: No such file or directory
>> tar: /etc/squid/squid.conf.pre.local: Cannot stat: No such file or 
>> directory
>>
>> There were seven lines in total like this and the three I checked were 
>> files that did not exist on my system. I presume that these are files 
>> that should be backed up if present but don't have to be present so 
>> that the error messages are nothing to worry about.
>> There were no error messages when doing the backup via the WUI.
>>
> 
> Good find. ;)
> Michael's patch removes the test of existence, globbing produces 
> existent file names only. But I suppose there are several full names of 
> files, not existing in each implementation.
> Conclusion: Repair the file globbing by the shopt and do the existence 
> check. So all existent files are chosen, that are listed in the include 
> files.
> 
> Regards,
> Bernhard
> 
>> The blank line was only on my vm testbed system but as I clone this 
>> for any testing activities I will have to check all of my existing 
>> vm's to fix this.
>>
>> Glad we resolved this simply. Sorry for the anxiety.
>>
>> Regards,
>>
>> Adolf.
>>
>>>
>>>>
>>>> Am 29.03.2022 um 16:14 schrieb Adolf Belka:
>>>>> Hi All,
>>>>> This is what I get with the pushd/popd version in place on the same 
>>>>> system
>>>>> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
>>>>> etc/group
>>>>> etc/hosts
>>>>> etc/hosts.allow
>>>>> etc/hosts.deny
>>>>> etc/httpd/server.crt
>>>>> etc/httpd/server.csr
>>>>> etc/httpd/server-ecdsa.crt
>>>>> etc/httpd/server-ecdsa.csr
>>>>> etc/httpd/server-ecdsa.key
>>>>> etc/httpd/server.key
>>>>> etc/ipsec.user.conf
>>>>> etc/ipsec.user.secrets
>>>>> etc/logrotate.d
>>>>> etc/passwd
>>>>> etc/shadow
>>>>> etc/ssh/sshd_config
>>>>> etc/ssh/ssh_host_ecdsa_key
>>>>> etc/ssh/ssh_host_ecdsa_key.pub
>>>>> etc/ssh/ssh_host_ed25519_key
>>>>> etc/ssh/ssh_host_ed25519_key.pub
>>>>> etc/ssh/ssh_host_rsa_key
>>>>> etc/ssh/ssh_host_rsa_key.pub
>>>>> /etc/sudoers
>>>>> etc/sysconfig/createfiles
>>>>> etc/sysconfig/firewall.local
>>>>> etc/sysconfig/lm_sensors
>>>>> etc/sysconfig/modules
>>>>> etc/sysconfig/ramdisk
>>>>> etc/sysconfig/rc
>>>>> etc/sysconfig/rc.local
>>>>> etc/unbound
>>>>> /home/ahb
>>>>> root/.bash_history
>>>>> var/ipfire/accounting/settings.conf
>>>>> var/ipfire/auth/users
>>>>> var/ipfire/backup/addons/backup
>>>>> var/ipfire/backup/exclude.user
>>>>> var/ipfire/backup/include.user
>>>>> var/ipfire/ca/cacert.pem
>>>>> var/ipfire/captive/agb.txt
>>>>> var/ipfire/captive/clients
>>>>> var/ipfire/captive/coupons
>>>>> var/ipfire/captive/logo.dat
>>>>> var/ipfire/captive/settings
>>>>> var/ipfire/captive/terms.txt
>>>>> var/ipfire/captive/voucher_out
>>>>> var/ipfire/certs/hostcert.pem
>>>>> var/ipfire/certs/hostkey.pem
>>>>> var/ipfire/certs/phoebevmipseccert.pem
>>>>> var/ipfire/connscheduler/connscheduler.conf
>>>>> var/ipfire/crls/cacrl.pem
>>>>> var/ipfire/cups/cups-browsed.conf
>>>>> var/ipfire/cups/subscriptions.conf
>>>>> var/ipfire/ddns/config
>>>>> var/ipfire/ddns/ddns.conf
>>>>> var/ipfire/ddns/settings
>>>>> var/ipfire/dhcp/advoptions
>>>>> var/ipfire/dhcp/advoptions-list
>>>>> var/ipfire/dhcpc/dhcpcd.conf
>>>>> var/ipfire/dhcp/dhcpd.conf
>>>>> var/ipfire/dhcp/dhcpd.conf.local
>>>>> var/ipfire/dhcp/enable_blue
>>>>> var/ipfire/dhcp/enable_green
>>>>> var/ipfire/dhcp/fixleases
>>>>> var/ipfire/dhcp/settings
>>>>> var/ipfire/dma/auth.conf
>>>>> var/ipfire/dma/dma.conf
>>>>> var/ipfire/dma/mail.conf
>>>>> var/ipfire/dns
>>>>> var/ipfire/dnsforward/config
>>>>> var/ipfire/dns/settings
>>>>> var/ipfire/ethernet/aliases
>>>>> var/ipfire/ethernet/settings
>>>>> var/ipfire/ethernet/wireless
>>>>> var/ipfire/extrahd/settings
>>>>> var/ipfire/firewall
>>>>> var/ipfire/firewall/config
>>>>> var/ipfire/firewall/settings
>>>>> var/ipfire/fwhosts
>>>>> var/ipfire/isdn/settings
>>>>> var/ipfire/logging/settings
>>>>> var/ipfire/mac/settings
>>>>> var/ipfire/main/firstsetup_ok
>>>>> var/ipfire/main/gpl_accepted
>>>>> var/ipfire/main/hostname.conf
>>>>> var/ipfire/main/hosts
>>>>> var/ipfire/main/manualpages
>>>>> var/ipfire/main/routing
>>>>> var/ipfire/main/security
>>>>> var/ipfire/main/send_profile
>>>>> var/ipfire/main/settings
>>>>> var/ipfire/modem/settings
>>>>> var/ipfire/optionsfw/settings
>>>>> var/ipfire/ovpn
>>>>> var/ipfire/ovpn/ccd.conf
>>>>> var/ipfire/ovpn/collectd.vpn
>>>>> var/ipfire/ovpn/enable
>>>>> var/ipfire/ovpn/server.conf
>>>>> var/ipfire/ovpn/settings
>>>>> var/ipfire/pakfire/settings
>>>>> var/ipfire/ppp
>>>>> var/ipfire/ppp/fake-resolv.conf
>>>>> var/ipfire/ppp/settings
>>>>> var/ipfire/private/cakey.pem
>>>>> var/ipfire/proxy
>>>>> var/ipfire/proxy/asnbl-helper.conf
>>>>> var/ipfire/proxy/cachemgr.conf
>>>>> var/ipfire/proxy/enable
>>>>> var/ipfire/proxy/settings
>>>>> var/ipfire/proxy/squid.conf
>>>>> var/ipfire/qos/bin
>>>>> var/ipfire/qos/classes
>>>>> var/ipfire/qos/level7config
>>>>> var/ipfire/qos/portconfig
>>>>> var/ipfire/qos/settings
>>>>> var/ipfire/qos/subclasses
>>>>> var/ipfire/qos/tosconfig
>>>>> var/ipfire/remote/enablessh
>>>>> var/ipfire/remote/settings
>>>>> var/ipfire/sensors/settings
>>>>> var/ipfire/suricata/oinkmaster.conf
>>>>> var/ipfire/suricata/oinkmaster-modify-sids.conf
>>>>> var/ipfire/suricata/oinkmaster-provider-includes.conf
>>>>> var/ipfire/suricata/providers-settings
>>>>> var/ipfire/suricata/settings
>>>>> var/ipfire/suricata/suricata-default-rules.yaml
>>>>> var/ipfire/suricata/suricata-dns-servers.yaml
>>>>> var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
>>>>> var/ipfire/suricata/suricata-homenet.yaml
>>>>> var/ipfire/suricata/suricata-http-ports.yaml
>>>>> var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml
>>>>> var/ipfire/suricata/suricata-used-providers.yaml
>>>>> var/ipfire/time/
>>>>> var/ipfire/time/counter.conf
>>>>> var/ipfire/time/enable
>>>>> var/ipfire/time/settime.conf
>>>>> var/ipfire/time/settings
>>>>> var/ipfire/upnp/settings
>>>>> var/ipfire/urlfilter
>>>>> var/ipfire/urlfilter/settings
>>>>> var/ipfire/urlfilter/squidGuard.conf
>>>>> var/ipfire/vpn
>>>>> var/ipfire/vpn/config
>>>>> var/ipfire/vpn/ipsec.conf
>>>>> var/ipfire/vpn/settings
>>>>> var/ipfire/wakeonlan/clients.conf
>>>>> var/ipfire/wio/wio.conf
>>>>> var/ipfire/wireless/config
>>>>> var/ipfire/wireless/settings
>>>>> var/lib/suricata
>>>>> var/log/rrd/collectd
>>>>> var/log/rrd/hddshutdown-md127.rrd
>>>>> var/log/rrd/hddshutdown-sda.rrd
>>>>> var/log/rrd/hddshutdown-sdb.rrd
>>>>> var/log/rrd/hddtemp-md127.rrd
>>>>> var/log/rrd/hddtemp-sda.rrd
>>>>> var/log/rrd/hddtemp-sdb.rrd
>>>>> var/log/rrd/wio
>>>>> var/log/vnstat
>>>>> var/tmp/idsrules-emerging.tar.gz
>>>>> var/tmp/idsrules-sslbl_blacklist.rules
>>>>> The following are in the previous list but not in this one:-
>>>>> /
>>>>> /etc/conntrackd/conntrackd.conf
>>>>> /etc/ipsec.user-post.conf
>>>>> /root/.gitconfig
>>>>> /root/.ssh
>>>>> Regards,
>>>>> Adolf.
>>>>> On 29/03/2022 16:01, Adolf Belka wrote:
>>>>>> Hi All,
>>>>>>
>>>>>> On 29/03/2022 15:36, Bernhard Bitsch wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> Am 29.03.2022 um 15:11 schrieb Michael Tremer:
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> Could you please send the file listing to find out what is being 
>>>>>>>> included what shouldn’t?
>>>>>> Unfortunately, as I stopped the backup continuing once it had 
>>>>>> reached 1.2GB, the file created was not able to be opened. 
>>>>>> Probably stopping the backup corrupted it in some way.
>>>>>>>>
>>>>>>>
>>>>>>> Just for easy quick handling, you can use
>>>>>>> '/var/ipfire/backup/bin/backup.pl list' to show the files included.
>>>>>> I gave this a go and here is the output from the command.
>>>>>>
>>>>>> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
>>>>>> /
>>>>>> /etc/conntrackd/conntrackd.conf
>>>>>> /etc/group
>>>>>> /etc/hosts
>>>>>> /etc/hosts.allow
>>>>>> /etc/hosts.deny
>>>>>> /etc/httpd/server.crt
>>>>>> /etc/httpd/server.csr
>>>>>> /etc/httpd/server-ecdsa.crt
>>>>>> /etc/httpd/server-ecdsa.csr
>>>>>> /etc/httpd/server-ecdsa.key
>>>>>> /etc/httpd/server.key
>>>>>> /etc/ipsec.user.conf
>>>>>> /etc/ipsec.user-post.conf
>>>>>> /etc/ipsec.user.secrets
>>>>>> /etc/logrotate.d
>>>>>> /etc/passwd
>>>>>> /etc/shadow
>>>>>> /etc/squid/squid.conf.local
>>>>>> /etc/squid/squid.conf.pre.local
>>>>>> /etc/ssh/sshd_config
>>>>>> /etc/ssh/ssh_host_ecdsa_key
>>>>>> /etc/ssh/ssh_host_ecdsa_key.pub
>>>>>> /etc/ssh/ssh_host_ed25519_key
>>>>>> /etc/ssh/ssh_host_ed25519_key.pub
>>>>>> /etc/ssh/ssh_host_rsa_key
>>>>>> /etc/ssh/ssh_host_rsa_key.pub
>>>>>> //etc/sudoers
>>>>>> /etc/sysconfig/createfiles
>>>>>> /etc/sysconfig/firewall.local
>>>>>> /etc/sysconfig/lm_sensors
>>>>>> /etc/sysconfig/modules
>>>>>> /etc/sysconfig/ramdisk
>>>>>> /etc/sysconfig/rc
>>>>>> /etc/sysconfig/rc.local
>>>>>> /etc/unbound
>>>>>> //home/ahb
>>>>>> /root/.bash_history
>>>>>> /root/.gitconfig
>>>>>> /root/.ssh
>>>>>> /var/ipfire/accounting/settings.conf
>>>>>> /var/ipfire/auth/users
>>>>>> /var/ipfire/backup/addons/backup
>>>>>> /var/ipfire/backup/exclude.user
>>>>>> /var/ipfire/backup/include.user
>>>>>> /var/ipfire/ca/cacert.pem
>>>>>> /var/ipfire/captive/agb.txt
>>>>>> /var/ipfire/captive/clients
>>>>>> /var/ipfire/captive/coupons
>>>>>> /var/ipfire/captive/logo.dat
>>>>>> /var/ipfire/captive/settings
>>>>>> /var/ipfire/captive/terms.txt
>>>>>> /var/ipfire/captive/voucher_out
>>>>>> /var/ipfire/certs/hostcert.pem
>>>>>> /var/ipfire/certs/hostkey.pem
>>>>>> /var/ipfire/certs/phoebevmipseccert.pem
>>>>>> /var/ipfire/connscheduler/connscheduler.conf
>>>>>> /var/ipfire/crls/cacrl.pem
>>>>>> /var/ipfire/cups/cups-browsed.conf
>>>>>> /var/ipfire/cups/subscriptions.conf
>>>>>> /var/ipfire/ddns/config
>>>>>> /var/ipfire/ddns/ddns.conf
>>>>>> /var/ipfire/ddns/settings
>>>>>> /var/ipfire/dhcp/advoptions
>>>>>> /var/ipfire/dhcp/advoptions-list
>>>>>> /var/ipfire/dhcpc/dhcpcd.conf
>>>>>> /var/ipfire/dhcp/dhcpd.conf
>>>>>> /var/ipfire/dhcp/dhcpd.conf.local
>>>>>> /var/ipfire/dhcp/enable_blue
>>>>>> /var/ipfire/dhcp/enable_green
>>>>>> /var/ipfire/dhcp/fixleases
>>>>>> /var/ipfire/dhcp/settings
>>>>>> /var/ipfire/dma/auth.conf
>>>>>> /var/ipfire/dma/dma.conf
>>>>>> /var/ipfire/dma/mail.conf
>>>>>> /var/ipfire/dns
>>>>>> /var/ipfire/dnsforward/config
>>>>>> /var/ipfire/dns/settings
>>>>>> /var/ipfire/ethernet/aliases
>>>>>> /var/ipfire/ethernet/settings
>>>>>> /var/ipfire/ethernet/wireless
>>>>>> /var/ipfire/extrahd/settings
>>>>>> /var/ipfire/firewall
>>>>>> /var/ipfire/firewall/config
>>>>>> /var/ipfire/firewall/settings
>>>>>> /var/ipfire/fwhosts
>>>>>> /var/ipfire/isdn/settings
>>>>>> /var/ipfire/logging/settings
>>>>>> /var/ipfire/mac/settings
>>>>>> /var/ipfire/main/firstsetup_ok
>>>>>> /var/ipfire/main/gpl_accepted
>>>>>> /var/ipfire/main/hostname.conf
>>>>>> /var/ipfire/main/hosts
>>>>>> /var/ipfire/main/manualpages
>>>>>> /var/ipfire/main/routing
>>>>>> /var/ipfire/main/security
>>>>>> /var/ipfire/main/send_profile
>>>>>> /var/ipfire/main/settings
>>>>>> /var/ipfire/modem/settings
>>>>>> /var/ipfire/optionsfw/settings
>>>>>> /var/ipfire/ovpn
>>>>>> /var/ipfire/ovpn/ccd.conf
>>>>>> /var/ipfire/ovpn/collectd.vpn
>>>>>> /var/ipfire/ovpn/enable
>>>>>> /var/ipfire/ovpn/server.conf
>>>>>> /var/ipfire/ovpn/settings
>>>>>> /var/ipfire/pakfire/settings
>>>>>> /var/ipfire/ppp
>>>>>> /var/ipfire/ppp/fake-resolv.conf
>>>>>> /var/ipfire/ppp/settings
>>>>>> /var/ipfire/private/cakey.pem
>>>>>> /var/ipfire/proxy
>>>>>> /var/ipfire/proxy/asnbl-helper.conf
>>>>>> /var/ipfire/proxy/cachemgr.conf
>>>>>> /var/ipfire/proxy/enable
>>>>>> /var/ipfire/proxy/settings
>>>>>> /var/ipfire/proxy/squid.conf
>>>>>> /var/ipfire/qos/bin
>>>>>> /var/ipfire/qos/bin/qos.sh
>>>>>> /var/ipfire/qos/classes
>>>>>> /var/ipfire/qos/level7config
>>>>>> /var/ipfire/qos/portconfig
>>>>>> /var/ipfire/qos/settings
>>>>>> /var/ipfire/qos/subclasses
>>>>>> /var/ipfire/qos/tosconfig
>>>>>> /var/ipfire/remote/enablessh
>>>>>> /var/ipfire/remote/settings
>>>>>> /var/ipfire/sensors/settings
>>>>>> /var/ipfire/suricata/oinkmaster.conf
>>>>>> /var/ipfire/suricata/oinkmaster-modify-sids.conf
>>>>>> /var/ipfire/suricata/oinkmaster-provider-includes.conf
>>>>>> /var/ipfire/suricata/providers-settings
>>>>>> /var/ipfire/suricata/settings
>>>>>> /var/ipfire/suricata/suricata-default-rules.yaml
>>>>>> /var/ipfire/suricata/suricata-dns-servers.yaml
>>>>>> /var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
>>>>>> /var/ipfire/suricata/suricata-homenet.yaml
>>>>>> /var/ipfire/suricata/suricata-http-ports.yaml
>>>>>> /var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml
>>>>>> /var/ipfire/suricata/suricata-used-providers.yaml
>>>>>> /var/ipfire/time/
>>>>>> /var/ipfire/time/counter.conf
>>>>>> /var/ipfire/time/enable
>>>>>> /var/ipfire/time/settime.conf
>>>>>> /var/ipfire/time/settings
>>>>>> /var/ipfire/upnp/settings
>>>>>> /var/ipfire/urlfilter
>>>>>> /var/ipfire/urlfilter/settings
>>>>>> /var/ipfire/urlfilter/squidGuard.conf
>>>>>> /var/ipfire/vpn
>>>>>> /var/ipfire/vpn/config
>>>>>> /var/ipfire/vpn/ipsec.conf
>>>>>> /var/ipfire/vpn/settings
>>>>>> /var/ipfire/wakeonlan/clients.conf
>>>>>> /var/ipfire/wio/wio.conf
>>>>>> /var/ipfire/wireless/config
>>>>>> /var/ipfire/wireless/settings
>>>>>> /var/lib/suricata
>>>>>> /var/log/rrd/collectd
>>>>>> /var/log/rrd/hddshutdown-md127.rrd
>>>>>> /var/log/rrd/hddshutdown-sda.rrd
>>>>>> /var/log/rrd/hddshutdown-sdb.rrd
>>>>>> /var/log/rrd/hddtemp-md127.rrd
>>>>>> /var/log/rrd/hddtemp-sda.rrd
>>>>>> /var/log/rrd/hddtemp-sdb.rrd
>>>>>> /var/log/rrd/wio
>>>>>> /var/log/vnstat
>>>>>> /var/tmp/idsrules-emerging.tar.gz
>>>>>> /var/tmp/idsrules-sslbl_blacklist.rules
>>>>>>
>>>>>> Regards,
>>>>>> Adolf.
>>>>>>>
>>>>>>> Bernhard
>>>>>>>> -Michael
>>>>>>>>
>>>>>>>>> On 29 Mar 2022, at 14:10, Adolf Belka <adolf.belka@ipfire.org> 
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Hi Michael,
>>>>>>>>>
>>>>>>>>> Just tried this patch out on my vm testbed system and it still 
>>>>>>>>> doesn't work for me. The backup file had got to 1.3GB when I 
>>>>>>>>> deleted the backup file as it was still growing. The normal 
>>>>>>>>> correct backup file on that vm machine is around 7MB
>>>>>>>>>
>>>>>>>>> The above was the case for both running it from the WUI or from 
>>>>>>>>> the command line from my unprivileged user using sudo 
>>>>>>>>> backupctrl exclude
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Adolf.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 29/03/2022 14:27, Michael Tremer wrote:
>>>>>>>>>> This patch fixes globbing expansion in the backup include file 
>>>>>>>>>> list
>>>>>>>>>> which got broken in c7e0d73e7cfd7be95db9d0a5f3392b8241813d5b.
>>>>>>>>>>
>>>>>>>>>> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
>>>>>>>>>> ---
>>>>>>>>>>    config/backup/backup.pl | 8 ++++----
>>>>>>>>>>    1 file changed, 4 insertions(+), 4 deletions(-)
>>>>>>>>>>
>>>>>>>>>> diff --git a/config/backup/backup.pl b/config/backup/backup.pl
>>>>>>>>>> index a2337cf23..6f9295e94 100644
>>>>>>>>>> --- a/config/backup/backup.pl
>>>>>>>>>> +++ b/config/backup/backup.pl
>>>>>>>>>> @@ -19,6 +19,8 @@
>>>>>>>>>> # #
>>>>>>>>>> ############################################################################### 
>>>>>>>>>>
>>>>>>>>>>    +shopt -s nullglob
>>>>>>>>>> +
>>>>>>>>>>    NOW="$(date "+%Y-%m-%d-%H:%M")"
>>>>>>>>>>      list_addons() {
>>>>>>>>>> @@ -38,10 +40,8 @@ process_includes() {
>>>>>>>>>>        for include in $@; do
>>>>>>>>>>            local file
>>>>>>>>>>            while read -r file; do
>>>>>>>>>> -            for file in ${file}; do
>>>>>>>>>> -                if [ -e "/${file}" ]; then
>>>>>>>>>> -                    echo "${file}"
>>>>>>>>>> -                fi
>>>>>>>>>> +            for file in /${file}; do
>>>>>>>>>> +                echo "${file}"
>>>>>>>>>>                done
>>>>>>>>>>            done < "${include}"
>>>>>>>>>>        done | sort -u
>>>>>>>>
>>>

Michael Tremer March 30, 2022, 2:06 p.m. UTC | #14

Hello everyone,

Thank you very much for helping me test this.

I decided that we will have to create an emergency core update for this, because it will otherwise break people’s system. We also have a vulnerability in zlib which is being handed around by the press, so that has helped me to make a decision.

I also pushed two more patches. One filters out any empty lines because they have quite a damaging effect and it happens easily that they are added to the include list. Secondly, I brought back the existence check which I thought was only there for the globbing mechanism, but also had the small side-effect of filtering out the file list early on which helps keeping tar happy.

Please review those changes and please install the new c166 from testing as soon as the build has finished (I am not sure if you are all on the nightly-builds list to see any notifications).

https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=4f0e7f24f293ca10096e616010cfd4417afc157c
https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=b275771fdd10ac7116e5d400dc10d3148df8ac27

Best,
-Michael

> On 29 Mar 2022, at 20:21, Bernhard Bitsch <bbitsch@ipfire.org> wrote:
> 
> Just checked the blank line problem.
> Pattern is empty , file name list is '/' --> no names produced ( with existence check )
> Pattern is empty --> produced file name '/' ( without -e check )
> 
> Am 29.03.2022 um 20:36 schrieb Bernhard Bitsch:
>> Hi all,
>> Am 29.03.2022 um 17:23 schrieb Adolf Belka:
>>> Hi All,
>>> 
>>> On 29/03/2022 16:22, Michael Tremer wrote:
>>>> 
>>>>> On 29 Mar 2022, at 15:22, Bernhard Bitsch <bbitsch@ipfire.org> wrote:
>>>>> 
>>>>> Looks like your include.user file contains absolute paths.
>>> That was not the problem but I have corrected that. I would have expected that when the main include and exclude files were made relative that the update script would also have checked for any entries in the include.user and exclude.user files but it looks like it didn't.th
>> According to the source they are chosen.
>>>> 
>>>> Or an empty line maybe?
>>> That turned out to be the problem. There was an empty line after the two lines I had added. You only saw it if you scrolled the pointer down the file in the editor.
>> Interesting. A blank line should produce nothing.
>>> 
>>> Removing that blank line removed the line only with a / and the backup then successfully ran and created a backup file containing all the directories expected.
>>> 
>>> The only thing I found is that there was the following tar error message
>>> 
>>> tar: Exiting with failure status due to previous errors
>>> 
>>> The only thing I found in the backup output was several lines such as
>>> 
>>> tar: /etc/squid/squid.conf.local: Cannot stat: No such file or directory
>>> tar: /etc/squid/squid.conf.pre.local: Cannot stat: No such file or directory
>>> 
>>> There were seven lines in total like this and the three I checked were files that did not exist on my system. I presume that these are files that should be backed up if present but don't have to be present so that the error messages are nothing to worry about.
>>> There were no error messages when doing the backup via the WUI.
>>> 
>> Good find. ;)
>> Michael's patch removes the test of existence, globbing produces existent file names only. But I suppose there are several full names of files, not existing in each implementation.
>> Conclusion: Repair the file globbing by the shopt and do the existence check. So all existent files are chosen, that are listed in the include files.
>> Regards,
>> Bernhard
>>> The blank line was only on my vm testbed system but as I clone this for any testing activities I will have to check all of my existing vm's to fix this.
>>> 
>>> Glad we resolved this simply. Sorry for the anxiety.
>>> 
>>> Regards,
>>> 
>>> Adolf.
>>> 
>>>> 
>>>>> 
>>>>> Am 29.03.2022 um 16:14 schrieb Adolf Belka:
>>>>>> Hi All,
>>>>>> This is what I get with the pushd/popd version in place on the same system
>>>>>> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
>>>>>> etc/group
>>>>>> etc/hosts
>>>>>> etc/hosts.allow
>>>>>> etc/hosts.deny
>>>>>> etc/httpd/server.crt
>>>>>> etc/httpd/server.csr
>>>>>> etc/httpd/server-ecdsa.crt
>>>>>> etc/httpd/server-ecdsa.csr
>>>>>> etc/httpd/server-ecdsa.key
>>>>>> etc/httpd/server.key
>>>>>> etc/ipsec.user.conf
>>>>>> etc/ipsec.user.secrets
>>>>>> etc/logrotate.d
>>>>>> etc/passwd
>>>>>> etc/shadow
>>>>>> etc/ssh/sshd_config
>>>>>> etc/ssh/ssh_host_ecdsa_key
>>>>>> etc/ssh/ssh_host_ecdsa_key.pub
>>>>>> etc/ssh/ssh_host_ed25519_key
>>>>>> etc/ssh/ssh_host_ed25519_key.pub
>>>>>> etc/ssh/ssh_host_rsa_key
>>>>>> etc/ssh/ssh_host_rsa_key.pub
>>>>>> /etc/sudoers
>>>>>> etc/sysconfig/createfiles
>>>>>> etc/sysconfig/firewall.local
>>>>>> etc/sysconfig/lm_sensors
>>>>>> etc/sysconfig/modules
>>>>>> etc/sysconfig/ramdisk
>>>>>> etc/sysconfig/rc
>>>>>> etc/sysconfig/rc.local
>>>>>> etc/unbound
>>>>>> /home/ahb
>>>>>> root/.bash_history
>>>>>> var/ipfire/accounting/settings.conf
>>>>>> var/ipfire/auth/users
>>>>>> var/ipfire/backup/addons/backup
>>>>>> var/ipfire/backup/exclude.user
>>>>>> var/ipfire/backup/include.user
>>>>>> var/ipfire/ca/cacert.pem
>>>>>> var/ipfire/captive/agb.txt
>>>>>> var/ipfire/captive/clients
>>>>>> var/ipfire/captive/coupons
>>>>>> var/ipfire/captive/logo.dat
>>>>>> var/ipfire/captive/settings
>>>>>> var/ipfire/captive/terms.txt
>>>>>> var/ipfire/captive/voucher_out
>>>>>> var/ipfire/certs/hostcert.pem
>>>>>> var/ipfire/certs/hostkey.pem
>>>>>> var/ipfire/certs/phoebevmipseccert.pem
>>>>>> var/ipfire/connscheduler/connscheduler.conf
>>>>>> var/ipfire/crls/cacrl.pem
>>>>>> var/ipfire/cups/cups-browsed.conf
>>>>>> var/ipfire/cups/subscriptions.conf
>>>>>> var/ipfire/ddns/config
>>>>>> var/ipfire/ddns/ddns.conf
>>>>>> var/ipfire/ddns/settings
>>>>>> var/ipfire/dhcp/advoptions
>>>>>> var/ipfire/dhcp/advoptions-list
>>>>>> var/ipfire/dhcpc/dhcpcd.conf
>>>>>> var/ipfire/dhcp/dhcpd.conf
>>>>>> var/ipfire/dhcp/dhcpd.conf.local
>>>>>> var/ipfire/dhcp/enable_blue
>>>>>> var/ipfire/dhcp/enable_green
>>>>>> var/ipfire/dhcp/fixleases
>>>>>> var/ipfire/dhcp/settings
>>>>>> var/ipfire/dma/auth.conf
>>>>>> var/ipfire/dma/dma.conf
>>>>>> var/ipfire/dma/mail.conf
>>>>>> var/ipfire/dns
>>>>>> var/ipfire/dnsforward/config
>>>>>> var/ipfire/dns/settings
>>>>>> var/ipfire/ethernet/aliases
>>>>>> var/ipfire/ethernet/settings
>>>>>> var/ipfire/ethernet/wireless
>>>>>> var/ipfire/extrahd/settings
>>>>>> var/ipfire/firewall
>>>>>> var/ipfire/firewall/config
>>>>>> var/ipfire/firewall/settings
>>>>>> var/ipfire/fwhosts
>>>>>> var/ipfire/isdn/settings
>>>>>> var/ipfire/logging/settings
>>>>>> var/ipfire/mac/settings
>>>>>> var/ipfire/main/firstsetup_ok
>>>>>> var/ipfire/main/gpl_accepted
>>>>>> var/ipfire/main/hostname.conf
>>>>>> var/ipfire/main/hosts
>>>>>> var/ipfire/main/manualpages
>>>>>> var/ipfire/main/routing
>>>>>> var/ipfire/main/security
>>>>>> var/ipfire/main/send_profile
>>>>>> var/ipfire/main/settings
>>>>>> var/ipfire/modem/settings
>>>>>> var/ipfire/optionsfw/settings
>>>>>> var/ipfire/ovpn
>>>>>> var/ipfire/ovpn/ccd.conf
>>>>>> var/ipfire/ovpn/collectd.vpn
>>>>>> var/ipfire/ovpn/enable
>>>>>> var/ipfire/ovpn/server.conf
>>>>>> var/ipfire/ovpn/settings
>>>>>> var/ipfire/pakfire/settings
>>>>>> var/ipfire/ppp
>>>>>> var/ipfire/ppp/fake-resolv.conf
>>>>>> var/ipfire/ppp/settings
>>>>>> var/ipfire/private/cakey.pem
>>>>>> var/ipfire/proxy
>>>>>> var/ipfire/proxy/asnbl-helper.conf
>>>>>> var/ipfire/proxy/cachemgr.conf
>>>>>> var/ipfire/proxy/enable
>>>>>> var/ipfire/proxy/settings
>>>>>> var/ipfire/proxy/squid.conf
>>>>>> var/ipfire/qos/bin
>>>>>> var/ipfire/qos/classes
>>>>>> var/ipfire/qos/level7config
>>>>>> var/ipfire/qos/portconfig
>>>>>> var/ipfire/qos/settings
>>>>>> var/ipfire/qos/subclasses
>>>>>> var/ipfire/qos/tosconfig
>>>>>> var/ipfire/remote/enablessh
>>>>>> var/ipfire/remote/settings
>>>>>> var/ipfire/sensors/settings
>>>>>> var/ipfire/suricata/oinkmaster.conf
>>>>>> var/ipfire/suricata/oinkmaster-modify-sids.conf
>>>>>> var/ipfire/suricata/oinkmaster-provider-includes.conf
>>>>>> var/ipfire/suricata/providers-settings
>>>>>> var/ipfire/suricata/settings
>>>>>> var/ipfire/suricata/suricata-default-rules.yaml
>>>>>> var/ipfire/suricata/suricata-dns-servers.yaml
>>>>>> var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
>>>>>> var/ipfire/suricata/suricata-homenet.yaml
>>>>>> var/ipfire/suricata/suricata-http-ports.yaml
>>>>>> var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml
>>>>>> var/ipfire/suricata/suricata-used-providers.yaml
>>>>>> var/ipfire/time/
>>>>>> var/ipfire/time/counter.conf
>>>>>> var/ipfire/time/enable
>>>>>> var/ipfire/time/settime.conf
>>>>>> var/ipfire/time/settings
>>>>>> var/ipfire/upnp/settings
>>>>>> var/ipfire/urlfilter
>>>>>> var/ipfire/urlfilter/settings
>>>>>> var/ipfire/urlfilter/squidGuard.conf
>>>>>> var/ipfire/vpn
>>>>>> var/ipfire/vpn/config
>>>>>> var/ipfire/vpn/ipsec.conf
>>>>>> var/ipfire/vpn/settings
>>>>>> var/ipfire/wakeonlan/clients.conf
>>>>>> var/ipfire/wio/wio.conf
>>>>>> var/ipfire/wireless/config
>>>>>> var/ipfire/wireless/settings
>>>>>> var/lib/suricata
>>>>>> var/log/rrd/collectd
>>>>>> var/log/rrd/hddshutdown-md127.rrd
>>>>>> var/log/rrd/hddshutdown-sda.rrd
>>>>>> var/log/rrd/hddshutdown-sdb.rrd
>>>>>> var/log/rrd/hddtemp-md127.rrd
>>>>>> var/log/rrd/hddtemp-sda.rrd
>>>>>> var/log/rrd/hddtemp-sdb.rrd
>>>>>> var/log/rrd/wio
>>>>>> var/log/vnstat
>>>>>> var/tmp/idsrules-emerging.tar.gz
>>>>>> var/tmp/idsrules-sslbl_blacklist.rules
>>>>>> The following are in the previous list but not in this one:-
>>>>>> /
>>>>>> /etc/conntrackd/conntrackd.conf
>>>>>> /etc/ipsec.user-post.conf
>>>>>> /root/.gitconfig
>>>>>> /root/.ssh
>>>>>> Regards,
>>>>>> Adolf.
>>>>>> On 29/03/2022 16:01, Adolf Belka wrote:
>>>>>>> Hi All,
>>>>>>> 
>>>>>>> On 29/03/2022 15:36, Bernhard Bitsch wrote:
>>>>>>>> Hi,
>>>>>>>> 
>>>>>>>> Am 29.03.2022 um 15:11 schrieb Michael Tremer:
>>>>>>>>> Hello,
>>>>>>>>> 
>>>>>>>>> Could you please send the file listing to find out what is being included what shouldn’t?
>>>>>>> Unfortunately, as I stopped the backup continuing once it had reached 1.2GB, the file created was not able to be opened. Probably stopping the backup corrupted it in some way.
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> Just for easy quick handling, you can use
>>>>>>>> '/var/ipfire/backup/bin/backup.pl list' to show the files included.
>>>>>>> I gave this a go and here is the output from the command.
>>>>>>> 
>>>>>>> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
>>>>>>> /
>>>>>>> /etc/conntrackd/conntrackd.conf
>>>>>>> /etc/group
>>>>>>> /etc/hosts
>>>>>>> /etc/hosts.allow
>>>>>>> /etc/hosts.deny
>>>>>>> /etc/httpd/server.crt
>>>>>>> /etc/httpd/server.csr
>>>>>>> /etc/httpd/server-ecdsa.crt
>>>>>>> /etc/httpd/server-ecdsa.csr
>>>>>>> /etc/httpd/server-ecdsa.key
>>>>>>> /etc/httpd/server.key
>>>>>>> /etc/ipsec.user.conf
>>>>>>> /etc/ipsec.user-post.conf
>>>>>>> /etc/ipsec.user.secrets
>>>>>>> /etc/logrotate.d
>>>>>>> /etc/passwd
>>>>>>> /etc/shadow
>>>>>>> /etc/squid/squid.conf.local
>>>>>>> /etc/squid/squid.conf.pre.local
>>>>>>> /etc/ssh/sshd_config
>>>>>>> /etc/ssh/ssh_host_ecdsa_key
>>>>>>> /etc/ssh/ssh_host_ecdsa_key.pub
>>>>>>> /etc/ssh/ssh_host_ed25519_key
>>>>>>> /etc/ssh/ssh_host_ed25519_key.pub
>>>>>>> /etc/ssh/ssh_host_rsa_key
>>>>>>> /etc/ssh/ssh_host_rsa_key.pub
>>>>>>> //etc/sudoers
>>>>>>> /etc/sysconfig/createfiles
>>>>>>> /etc/sysconfig/firewall.local
>>>>>>> /etc/sysconfig/lm_sensors
>>>>>>> /etc/sysconfig/modules
>>>>>>> /etc/sysconfig/ramdisk
>>>>>>> /etc/sysconfig/rc
>>>>>>> /etc/sysconfig/rc.local
>>>>>>> /etc/unbound
>>>>>>> //home/ahb
>>>>>>> /root/.bash_history
>>>>>>> /root/.gitconfig
>>>>>>> /root/.ssh
>>>>>>> /var/ipfire/accounting/settings.conf
>>>>>>> /var/ipfire/auth/users
>>>>>>> /var/ipfire/backup/addons/backup
>>>>>>> /var/ipfire/backup/exclude.user
>>>>>>> /var/ipfire/backup/include.user
>>>>>>> /var/ipfire/ca/cacert.pem
>>>>>>> /var/ipfire/captive/agb.txt
>>>>>>> /var/ipfire/captive/clients
>>>>>>> /var/ipfire/captive/coupons
>>>>>>> /var/ipfire/captive/logo.dat
>>>>>>> /var/ipfire/captive/settings
>>>>>>> /var/ipfire/captive/terms.txt
>>>>>>> /var/ipfire/captive/voucher_out
>>>>>>> /var/ipfire/certs/hostcert.pem
>>>>>>> /var/ipfire/certs/hostkey.pem
>>>>>>> /var/ipfire/certs/phoebevmipseccert.pem
>>>>>>> /var/ipfire/connscheduler/connscheduler.conf
>>>>>>> /var/ipfire/crls/cacrl.pem
>>>>>>> /var/ipfire/cups/cups-browsed.conf
>>>>>>> /var/ipfire/cups/subscriptions.conf
>>>>>>> /var/ipfire/ddns/config
>>>>>>> /var/ipfire/ddns/ddns.conf
>>>>>>> /var/ipfire/ddns/settings
>>>>>>> /var/ipfire/dhcp/advoptions
>>>>>>> /var/ipfire/dhcp/advoptions-list
>>>>>>> /var/ipfire/dhcpc/dhcpcd.conf
>>>>>>> /var/ipfire/dhcp/dhcpd.conf
>>>>>>> /var/ipfire/dhcp/dhcpd.conf.local
>>>>>>> /var/ipfire/dhcp/enable_blue
>>>>>>> /var/ipfire/dhcp/enable_green
>>>>>>> /var/ipfire/dhcp/fixleases
>>>>>>> /var/ipfire/dhcp/settings
>>>>>>> /var/ipfire/dma/auth.conf
>>>>>>> /var/ipfire/dma/dma.conf
>>>>>>> /var/ipfire/dma/mail.conf
>>>>>>> /var/ipfire/dns
>>>>>>> /var/ipfire/dnsforward/config
>>>>>>> /var/ipfire/dns/settings
>>>>>>> /var/ipfire/ethernet/aliases
>>>>>>> /var/ipfire/ethernet/settings
>>>>>>> /var/ipfire/ethernet/wireless
>>>>>>> /var/ipfire/extrahd/settings
>>>>>>> /var/ipfire/firewall
>>>>>>> /var/ipfire/firewall/config
>>>>>>> /var/ipfire/firewall/settings
>>>>>>> /var/ipfire/fwhosts
>>>>>>> /var/ipfire/isdn/settings
>>>>>>> /var/ipfire/logging/settings
>>>>>>> /var/ipfire/mac/settings
>>>>>>> /var/ipfire/main/firstsetup_ok
>>>>>>> /var/ipfire/main/gpl_accepted
>>>>>>> /var/ipfire/main/hostname.conf
>>>>>>> /var/ipfire/main/hosts
>>>>>>> /var/ipfire/main/manualpages
>>>>>>> /var/ipfire/main/routing
>>>>>>> /var/ipfire/main/security
>>>>>>> /var/ipfire/main/send_profile
>>>>>>> /var/ipfire/main/settings
>>>>>>> /var/ipfire/modem/settings
>>>>>>> /var/ipfire/optionsfw/settings
>>>>>>> /var/ipfire/ovpn
>>>>>>> /var/ipfire/ovpn/ccd.conf
>>>>>>> /var/ipfire/ovpn/collectd.vpn
>>>>>>> /var/ipfire/ovpn/enable
>>>>>>> /var/ipfire/ovpn/server.conf
>>>>>>> /var/ipfire/ovpn/settings
>>>>>>> /var/ipfire/pakfire/settings
>>>>>>> /var/ipfire/ppp
>>>>>>> /var/ipfire/ppp/fake-resolv.conf
>>>>>>> /var/ipfire/ppp/settings
>>>>>>> /var/ipfire/private/cakey.pem
>>>>>>> /var/ipfire/proxy
>>>>>>> /var/ipfire/proxy/asnbl-helper.conf
>>>>>>> /var/ipfire/proxy/cachemgr.conf
>>>>>>> /var/ipfire/proxy/enable
>>>>>>> /var/ipfire/proxy/settings
>>>>>>> /var/ipfire/proxy/squid.conf
>>>>>>> /var/ipfire/qos/bin
>>>>>>> /var/ipfire/qos/bin/qos.sh
>>>>>>> /var/ipfire/qos/classes
>>>>>>> /var/ipfire/qos/level7config
>>>>>>> /var/ipfire/qos/portconfig
>>>>>>> /var/ipfire/qos/settings
>>>>>>> /var/ipfire/qos/subclasses
>>>>>>> /var/ipfire/qos/tosconfig
>>>>>>> /var/ipfire/remote/enablessh
>>>>>>> /var/ipfire/remote/settings
>>>>>>> /var/ipfire/sensors/settings
>>>>>>> /var/ipfire/suricata/oinkmaster.conf
>>>>>>> /var/ipfire/suricata/oinkmaster-modify-sids.conf
>>>>>>> /var/ipfire/suricata/oinkmaster-provider-includes.conf
>>>>>>> /var/ipfire/suricata/providers-settings
>>>>>>> /var/ipfire/suricata/settings
>>>>>>> /var/ipfire/suricata/suricata-default-rules.yaml
>>>>>>> /var/ipfire/suricata/suricata-dns-servers.yaml
>>>>>>> /var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
>>>>>>> /var/ipfire/suricata/suricata-homenet.yaml
>>>>>>> /var/ipfire/suricata/suricata-http-ports.yaml
>>>>>>> /var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml
>>>>>>> /var/ipfire/suricata/suricata-used-providers.yaml
>>>>>>> /var/ipfire/time/
>>>>>>> /var/ipfire/time/counter.conf
>>>>>>> /var/ipfire/time/enable
>>>>>>> /var/ipfire/time/settime.conf
>>>>>>> /var/ipfire/time/settings
>>>>>>> /var/ipfire/upnp/settings
>>>>>>> /var/ipfire/urlfilter
>>>>>>> /var/ipfire/urlfilter/settings
>>>>>>> /var/ipfire/urlfilter/squidGuard.conf
>>>>>>> /var/ipfire/vpn
>>>>>>> /var/ipfire/vpn/config
>>>>>>> /var/ipfire/vpn/ipsec.conf
>>>>>>> /var/ipfire/vpn/settings
>>>>>>> /var/ipfire/wakeonlan/clients.conf
>>>>>>> /var/ipfire/wio/wio.conf
>>>>>>> /var/ipfire/wireless/config
>>>>>>> /var/ipfire/wireless/settings
>>>>>>> /var/lib/suricata
>>>>>>> /var/log/rrd/collectd
>>>>>>> /var/log/rrd/hddshutdown-md127.rrd
>>>>>>> /var/log/rrd/hddshutdown-sda.rrd
>>>>>>> /var/log/rrd/hddshutdown-sdb.rrd
>>>>>>> /var/log/rrd/hddtemp-md127.rrd
>>>>>>> /var/log/rrd/hddtemp-sda.rrd
>>>>>>> /var/log/rrd/hddtemp-sdb.rrd
>>>>>>> /var/log/rrd/wio
>>>>>>> /var/log/vnstat
>>>>>>> /var/tmp/idsrules-emerging.tar.gz
>>>>>>> /var/tmp/idsrules-sslbl_blacklist.rules
>>>>>>> 
>>>>>>> Regards,
>>>>>>> Adolf.
>>>>>>>> 
>>>>>>>> Bernhard
>>>>>>>>> -Michael
>>>>>>>>> 
>>>>>>>>>> On 29 Mar 2022, at 14:10, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>>>>>>>>> 
>>>>>>>>>> Hi Michael,
>>>>>>>>>> 
>>>>>>>>>> Just tried this patch out on my vm testbed system and it still doesn't work for me. The backup file had got to 1.3GB when I deleted the backup file as it was still growing. The normal correct backup file on that vm machine is around 7MB
>>>>>>>>>> 
>>>>>>>>>> The above was the case for both running it from the WUI or from the command line from my unprivileged user using sudo backupctrl exclude
>>>>>>>>>> 
>>>>>>>>>> Regards,
>>>>>>>>>> 
>>>>>>>>>> Adolf.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On 29/03/2022 14:27, Michael Tremer wrote:
>>>>>>>>>>> This patch fixes globbing expansion in the backup include file list
>>>>>>>>>>> which got broken in c7e0d73e7cfd7be95db9d0a5f3392b8241813d5b.
>>>>>>>>>>> 
>>>>>>>>>>> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
>>>>>>>>>>> ---
>>>>>>>>>>>    config/backup/backup.pl | 8 ++++----
>>>>>>>>>>>    1 file changed, 4 insertions(+), 4 deletions(-)
>>>>>>>>>>> 
>>>>>>>>>>> diff --git a/config/backup/backup.pl b/config/backup/backup.pl
>>>>>>>>>>> index a2337cf23..6f9295e94 100644
>>>>>>>>>>> --- a/config/backup/backup.pl
>>>>>>>>>>> +++ b/config/backup/backup.pl
>>>>>>>>>>> @@ -19,6 +19,8 @@
>>>>>>>>>>> # #
>>>>>>>>>>> ############################################################################### 
>>>>>>>>>>>    +shopt -s nullglob
>>>>>>>>>>> +
>>>>>>>>>>>    NOW="$(date "+%Y-%m-%d-%H:%M")"
>>>>>>>>>>>      list_addons() {
>>>>>>>>>>> @@ -38,10 +40,8 @@ process_includes() {
>>>>>>>>>>>        for include in $@; do
>>>>>>>>>>>            local file
>>>>>>>>>>>            while read -r file; do
>>>>>>>>>>> -            for file in ${file}; do
>>>>>>>>>>> -                if [ -e "/${file}" ]; then
>>>>>>>>>>> -                    echo "${file}"
>>>>>>>>>>> -                fi
>>>>>>>>>>> +            for file in /${file}; do
>>>>>>>>>>> +                echo "${file}"
>>>>>>>>>>>                done
>>>>>>>>>>>            done < "${include}"
>>>>>>>>>>>        done | sort -u
>>>>>>>>> 
>>>>

Bernhard Bitsch March 30, 2022, 2:38 p.m. UTC | #15

Hello all,


Am 30.03.2022 um 16:06 schrieb Michael Tremer:
> Hello everyone,
> 
> Thank you very much for helping me test this.
> 
> I decided that we will have to create an emergency core update for this, because it will otherwise break people’s system. We also have a vulnerability in zlib which is being handed around by the press, so that has helped me to make a decision.
> 

I think this is a good solution. Shall we add a warning to backup wiki 
page? Because in the moment a backup -> reinstall -> restore process 
leaves an incomplete system.

> I also pushed two more patches. One filters out any empty lines because they have quite a damaging effect and it happens easily that they are added to the include list. Secondly, I brought back the existence check which I thought was only there for the globbing mechanism, but also had the small side-effect of filtering out the file list early on which helps keeping tar happy.
> 
> Please review those changes and please install the new c166 from testing as soon as the build has finished (I am not sure if you are all on the nightly-builds list to see any notifications).
> 
> https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=4f0e7f24f293ca10096e616010cfd4417afc157c
The test on empty string should be in the 'while read -r file' loop. The 
empty lines are in the include files. The outer for loop iterates over 
the file list.

Regards,
Bernhard

> https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=b275771fdd10ac7116e5d400dc10d3148df8ac27
> 
> Best,
> -Michael
> 
>> On 29 Mar 2022, at 20:21, Bernhard Bitsch <bbitsch@ipfire.org> wrote:
>>
>> Just checked the blank line problem.
>> Pattern is empty , file name list is '/' --> no names produced ( with existence check )
>> Pattern is empty --> produced file name '/' ( without -e check )
>>
>> Am 29.03.2022 um 20:36 schrieb Bernhard Bitsch:
>>> Hi all,
>>> Am 29.03.2022 um 17:23 schrieb Adolf Belka:
>>>> Hi All,
>>>>
>>>> On 29/03/2022 16:22, Michael Tremer wrote:
>>>>>
>>>>>> On 29 Mar 2022, at 15:22, Bernhard Bitsch <bbitsch@ipfire.org> wrote:
>>>>>>
>>>>>> Looks like your include.user file contains absolute paths.
>>>> That was not the problem but I have corrected that. I would have expected that when the main include and exclude files were made relative that the update script would also have checked for any entries in the include.user and exclude.user files but it looks like it didn't.th
>>> According to the source they are chosen.
>>>>>
>>>>> Or an empty line maybe?
>>>> That turned out to be the problem. There was an empty line after the two lines I had added. You only saw it if you scrolled the pointer down the file in the editor.
>>> Interesting. A blank line should produce nothing.
>>>>
>>>> Removing that blank line removed the line only with a / and the backup then successfully ran and created a backup file containing all the directories expected.
>>>>
>>>> The only thing I found is that there was the following tar error message
>>>>
>>>> tar: Exiting with failure status due to previous errors
>>>>
>>>> The only thing I found in the backup output was several lines such as
>>>>
>>>> tar: /etc/squid/squid.conf.local: Cannot stat: No such file or directory
>>>> tar: /etc/squid/squid.conf.pre.local: Cannot stat: No such file or directory
>>>>
>>>> There were seven lines in total like this and the three I checked were files that did not exist on my system. I presume that these are files that should be backed up if present but don't have to be present so that the error messages are nothing to worry about.
>>>> There were no error messages when doing the backup via the WUI.
>>>>
>>> Good find. ;)
>>> Michael's patch removes the test of existence, globbing produces existent file names only. But I suppose there are several full names of files, not existing in each implementation.
>>> Conclusion: Repair the file globbing by the shopt and do the existence check. So all existent files are chosen, that are listed in the include files.
>>> Regards,
>>> Bernhard
>>>> The blank line was only on my vm testbed system but as I clone this for any testing activities I will have to check all of my existing vm's to fix this.
>>>>
>>>> Glad we resolved this simply. Sorry for the anxiety.
>>>>
>>>> Regards,
>>>>
>>>> Adolf.
>>>>
>>>>>
>>>>>>
>>>>>> Am 29.03.2022 um 16:14 schrieb Adolf Belka:
>>>>>>> Hi All,
>>>>>>> This is what I get with the pushd/popd version in place on the same system
>>>>>>> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
>>>>>>> etc/group
>>>>>>> etc/hosts
>>>>>>> etc/hosts.allow
>>>>>>> etc/hosts.deny
>>>>>>> etc/httpd/server.crt
>>>>>>> etc/httpd/server.csr
>>>>>>> etc/httpd/server-ecdsa.crt
>>>>>>> etc/httpd/server-ecdsa.csr
>>>>>>> etc/httpd/server-ecdsa.key
>>>>>>> etc/httpd/server.key
>>>>>>> etc/ipsec.user.conf
>>>>>>> etc/ipsec.user.secrets
>>>>>>> etc/logrotate.d
>>>>>>> etc/passwd
>>>>>>> etc/shadow
>>>>>>> etc/ssh/sshd_config
>>>>>>> etc/ssh/ssh_host_ecdsa_key
>>>>>>> etc/ssh/ssh_host_ecdsa_key.pub
>>>>>>> etc/ssh/ssh_host_ed25519_key
>>>>>>> etc/ssh/ssh_host_ed25519_key.pub
>>>>>>> etc/ssh/ssh_host_rsa_key
>>>>>>> etc/ssh/ssh_host_rsa_key.pub
>>>>>>> /etc/sudoers
>>>>>>> etc/sysconfig/createfiles
>>>>>>> etc/sysconfig/firewall.local
>>>>>>> etc/sysconfig/lm_sensors
>>>>>>> etc/sysconfig/modules
>>>>>>> etc/sysconfig/ramdisk
>>>>>>> etc/sysconfig/rc
>>>>>>> etc/sysconfig/rc.local
>>>>>>> etc/unbound
>>>>>>> /home/ahb
>>>>>>> root/.bash_history
>>>>>>> var/ipfire/accounting/settings.conf
>>>>>>> var/ipfire/auth/users
>>>>>>> var/ipfire/backup/addons/backup
>>>>>>> var/ipfire/backup/exclude.user
>>>>>>> var/ipfire/backup/include.user
>>>>>>> var/ipfire/ca/cacert.pem
>>>>>>> var/ipfire/captive/agb.txt
>>>>>>> var/ipfire/captive/clients
>>>>>>> var/ipfire/captive/coupons
>>>>>>> var/ipfire/captive/logo.dat
>>>>>>> var/ipfire/captive/settings
>>>>>>> var/ipfire/captive/terms.txt
>>>>>>> var/ipfire/captive/voucher_out
>>>>>>> var/ipfire/certs/hostcert.pem
>>>>>>> var/ipfire/certs/hostkey.pem
>>>>>>> var/ipfire/certs/phoebevmipseccert.pem
>>>>>>> var/ipfire/connscheduler/connscheduler.conf
>>>>>>> var/ipfire/crls/cacrl.pem
>>>>>>> var/ipfire/cups/cups-browsed.conf
>>>>>>> var/ipfire/cups/subscriptions.conf
>>>>>>> var/ipfire/ddns/config
>>>>>>> var/ipfire/ddns/ddns.conf
>>>>>>> var/ipfire/ddns/settings
>>>>>>> var/ipfire/dhcp/advoptions
>>>>>>> var/ipfire/dhcp/advoptions-list
>>>>>>> var/ipfire/dhcpc/dhcpcd.conf
>>>>>>> var/ipfire/dhcp/dhcpd.conf
>>>>>>> var/ipfire/dhcp/dhcpd.conf.local
>>>>>>> var/ipfire/dhcp/enable_blue
>>>>>>> var/ipfire/dhcp/enable_green
>>>>>>> var/ipfire/dhcp/fixleases
>>>>>>> var/ipfire/dhcp/settings
>>>>>>> var/ipfire/dma/auth.conf
>>>>>>> var/ipfire/dma/dma.conf
>>>>>>> var/ipfire/dma/mail.conf
>>>>>>> var/ipfire/dns
>>>>>>> var/ipfire/dnsforward/config
>>>>>>> var/ipfire/dns/settings
>>>>>>> var/ipfire/ethernet/aliases
>>>>>>> var/ipfire/ethernet/settings
>>>>>>> var/ipfire/ethernet/wireless
>>>>>>> var/ipfire/extrahd/settings
>>>>>>> var/ipfire/firewall
>>>>>>> var/ipfire/firewall/config
>>>>>>> var/ipfire/firewall/settings
>>>>>>> var/ipfire/fwhosts
>>>>>>> var/ipfire/isdn/settings
>>>>>>> var/ipfire/logging/settings
>>>>>>> var/ipfire/mac/settings
>>>>>>> var/ipfire/main/firstsetup_ok
>>>>>>> var/ipfire/main/gpl_accepted
>>>>>>> var/ipfire/main/hostname.conf
>>>>>>> var/ipfire/main/hosts
>>>>>>> var/ipfire/main/manualpages
>>>>>>> var/ipfire/main/routing
>>>>>>> var/ipfire/main/security
>>>>>>> var/ipfire/main/send_profile
>>>>>>> var/ipfire/main/settings
>>>>>>> var/ipfire/modem/settings
>>>>>>> var/ipfire/optionsfw/settings
>>>>>>> var/ipfire/ovpn
>>>>>>> var/ipfire/ovpn/ccd.conf
>>>>>>> var/ipfire/ovpn/collectd.vpn
>>>>>>> var/ipfire/ovpn/enable
>>>>>>> var/ipfire/ovpn/server.conf
>>>>>>> var/ipfire/ovpn/settings
>>>>>>> var/ipfire/pakfire/settings
>>>>>>> var/ipfire/ppp
>>>>>>> var/ipfire/ppp/fake-resolv.conf
>>>>>>> var/ipfire/ppp/settings
>>>>>>> var/ipfire/private/cakey.pem
>>>>>>> var/ipfire/proxy
>>>>>>> var/ipfire/proxy/asnbl-helper.conf
>>>>>>> var/ipfire/proxy/cachemgr.conf
>>>>>>> var/ipfire/proxy/enable
>>>>>>> var/ipfire/proxy/settings
>>>>>>> var/ipfire/proxy/squid.conf
>>>>>>> var/ipfire/qos/bin
>>>>>>> var/ipfire/qos/classes
>>>>>>> var/ipfire/qos/level7config
>>>>>>> var/ipfire/qos/portconfig
>>>>>>> var/ipfire/qos/settings
>>>>>>> var/ipfire/qos/subclasses
>>>>>>> var/ipfire/qos/tosconfig
>>>>>>> var/ipfire/remote/enablessh
>>>>>>> var/ipfire/remote/settings
>>>>>>> var/ipfire/sensors/settings
>>>>>>> var/ipfire/suricata/oinkmaster.conf
>>>>>>> var/ipfire/suricata/oinkmaster-modify-sids.conf
>>>>>>> var/ipfire/suricata/oinkmaster-provider-includes.conf
>>>>>>> var/ipfire/suricata/providers-settings
>>>>>>> var/ipfire/suricata/settings
>>>>>>> var/ipfire/suricata/suricata-default-rules.yaml
>>>>>>> var/ipfire/suricata/suricata-dns-servers.yaml
>>>>>>> var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
>>>>>>> var/ipfire/suricata/suricata-homenet.yaml
>>>>>>> var/ipfire/suricata/suricata-http-ports.yaml
>>>>>>> var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml
>>>>>>> var/ipfire/suricata/suricata-used-providers.yaml
>>>>>>> var/ipfire/time/
>>>>>>> var/ipfire/time/counter.conf
>>>>>>> var/ipfire/time/enable
>>>>>>> var/ipfire/time/settime.conf
>>>>>>> var/ipfire/time/settings
>>>>>>> var/ipfire/upnp/settings
>>>>>>> var/ipfire/urlfilter
>>>>>>> var/ipfire/urlfilter/settings
>>>>>>> var/ipfire/urlfilter/squidGuard.conf
>>>>>>> var/ipfire/vpn
>>>>>>> var/ipfire/vpn/config
>>>>>>> var/ipfire/vpn/ipsec.conf
>>>>>>> var/ipfire/vpn/settings
>>>>>>> var/ipfire/wakeonlan/clients.conf
>>>>>>> var/ipfire/wio/wio.conf
>>>>>>> var/ipfire/wireless/config
>>>>>>> var/ipfire/wireless/settings
>>>>>>> var/lib/suricata
>>>>>>> var/log/rrd/collectd
>>>>>>> var/log/rrd/hddshutdown-md127.rrd
>>>>>>> var/log/rrd/hddshutdown-sda.rrd
>>>>>>> var/log/rrd/hddshutdown-sdb.rrd
>>>>>>> var/log/rrd/hddtemp-md127.rrd
>>>>>>> var/log/rrd/hddtemp-sda.rrd
>>>>>>> var/log/rrd/hddtemp-sdb.rrd
>>>>>>> var/log/rrd/wio
>>>>>>> var/log/vnstat
>>>>>>> var/tmp/idsrules-emerging.tar.gz
>>>>>>> var/tmp/idsrules-sslbl_blacklist.rules
>>>>>>> The following are in the previous list but not in this one:-
>>>>>>> /
>>>>>>> /etc/conntrackd/conntrackd.conf
>>>>>>> /etc/ipsec.user-post.conf
>>>>>>> /root/.gitconfig
>>>>>>> /root/.ssh
>>>>>>> Regards,
>>>>>>> Adolf.
>>>>>>> On 29/03/2022 16:01, Adolf Belka wrote:
>>>>>>>> Hi All,
>>>>>>>>
>>>>>>>> On 29/03/2022 15:36, Bernhard Bitsch wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> Am 29.03.2022 um 15:11 schrieb Michael Tremer:
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>> Could you please send the file listing to find out what is being included what shouldn’t?
>>>>>>>> Unfortunately, as I stopped the backup continuing once it had reached 1.2GB, the file created was not able to be opened. Probably stopping the backup corrupted it in some way.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Just for easy quick handling, you can use
>>>>>>>>> '/var/ipfire/backup/bin/backup.pl list' to show the files included.
>>>>>>>> I gave this a go and here is the output from the command.
>>>>>>>>
>>>>>>>> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
>>>>>>>> /
>>>>>>>> /etc/conntrackd/conntrackd.conf
>>>>>>>> /etc/group
>>>>>>>> /etc/hosts
>>>>>>>> /etc/hosts.allow
>>>>>>>> /etc/hosts.deny
>>>>>>>> /etc/httpd/server.crt
>>>>>>>> /etc/httpd/server.csr
>>>>>>>> /etc/httpd/server-ecdsa.crt
>>>>>>>> /etc/httpd/server-ecdsa.csr
>>>>>>>> /etc/httpd/server-ecdsa.key
>>>>>>>> /etc/httpd/server.key
>>>>>>>> /etc/ipsec.user.conf
>>>>>>>> /etc/ipsec.user-post.conf
>>>>>>>> /etc/ipsec.user.secrets
>>>>>>>> /etc/logrotate.d
>>>>>>>> /etc/passwd
>>>>>>>> /etc/shadow
>>>>>>>> /etc/squid/squid.conf.local
>>>>>>>> /etc/squid/squid.conf.pre.local
>>>>>>>> /etc/ssh/sshd_config
>>>>>>>> /etc/ssh/ssh_host_ecdsa_key
>>>>>>>> /etc/ssh/ssh_host_ecdsa_key.pub
>>>>>>>> /etc/ssh/ssh_host_ed25519_key
>>>>>>>> /etc/ssh/ssh_host_ed25519_key.pub
>>>>>>>> /etc/ssh/ssh_host_rsa_key
>>>>>>>> /etc/ssh/ssh_host_rsa_key.pub
>>>>>>>> //etc/sudoers
>>>>>>>> /etc/sysconfig/createfiles
>>>>>>>> /etc/sysconfig/firewall.local
>>>>>>>> /etc/sysconfig/lm_sensors
>>>>>>>> /etc/sysconfig/modules
>>>>>>>> /etc/sysconfig/ramdisk
>>>>>>>> /etc/sysconfig/rc
>>>>>>>> /etc/sysconfig/rc.local
>>>>>>>> /etc/unbound
>>>>>>>> //home/ahb
>>>>>>>> /root/.bash_history
>>>>>>>> /root/.gitconfig
>>>>>>>> /root/.ssh
>>>>>>>> /var/ipfire/accounting/settings.conf
>>>>>>>> /var/ipfire/auth/users
>>>>>>>> /var/ipfire/backup/addons/backup
>>>>>>>> /var/ipfire/backup/exclude.user
>>>>>>>> /var/ipfire/backup/include.user
>>>>>>>> /var/ipfire/ca/cacert.pem
>>>>>>>> /var/ipfire/captive/agb.txt
>>>>>>>> /var/ipfire/captive/clients
>>>>>>>> /var/ipfire/captive/coupons
>>>>>>>> /var/ipfire/captive/logo.dat
>>>>>>>> /var/ipfire/captive/settings
>>>>>>>> /var/ipfire/captive/terms.txt
>>>>>>>> /var/ipfire/captive/voucher_out
>>>>>>>> /var/ipfire/certs/hostcert.pem
>>>>>>>> /var/ipfire/certs/hostkey.pem
>>>>>>>> /var/ipfire/certs/phoebevmipseccert.pem
>>>>>>>> /var/ipfire/connscheduler/connscheduler.conf
>>>>>>>> /var/ipfire/crls/cacrl.pem
>>>>>>>> /var/ipfire/cups/cups-browsed.conf
>>>>>>>> /var/ipfire/cups/subscriptions.conf
>>>>>>>> /var/ipfire/ddns/config
>>>>>>>> /var/ipfire/ddns/ddns.conf
>>>>>>>> /var/ipfire/ddns/settings
>>>>>>>> /var/ipfire/dhcp/advoptions
>>>>>>>> /var/ipfire/dhcp/advoptions-list
>>>>>>>> /var/ipfire/dhcpc/dhcpcd.conf
>>>>>>>> /var/ipfire/dhcp/dhcpd.conf
>>>>>>>> /var/ipfire/dhcp/dhcpd.conf.local
>>>>>>>> /var/ipfire/dhcp/enable_blue
>>>>>>>> /var/ipfire/dhcp/enable_green
>>>>>>>> /var/ipfire/dhcp/fixleases
>>>>>>>> /var/ipfire/dhcp/settings
>>>>>>>> /var/ipfire/dma/auth.conf
>>>>>>>> /var/ipfire/dma/dma.conf
>>>>>>>> /var/ipfire/dma/mail.conf
>>>>>>>> /var/ipfire/dns
>>>>>>>> /var/ipfire/dnsforward/config
>>>>>>>> /var/ipfire/dns/settings
>>>>>>>> /var/ipfire/ethernet/aliases
>>>>>>>> /var/ipfire/ethernet/settings
>>>>>>>> /var/ipfire/ethernet/wireless
>>>>>>>> /var/ipfire/extrahd/settings
>>>>>>>> /var/ipfire/firewall
>>>>>>>> /var/ipfire/firewall/config
>>>>>>>> /var/ipfire/firewall/settings
>>>>>>>> /var/ipfire/fwhosts
>>>>>>>> /var/ipfire/isdn/settings
>>>>>>>> /var/ipfire/logging/settings
>>>>>>>> /var/ipfire/mac/settings
>>>>>>>> /var/ipfire/main/firstsetup_ok
>>>>>>>> /var/ipfire/main/gpl_accepted
>>>>>>>> /var/ipfire/main/hostname.conf
>>>>>>>> /var/ipfire/main/hosts
>>>>>>>> /var/ipfire/main/manualpages
>>>>>>>> /var/ipfire/main/routing
>>>>>>>> /var/ipfire/main/security
>>>>>>>> /var/ipfire/main/send_profile
>>>>>>>> /var/ipfire/main/settings
>>>>>>>> /var/ipfire/modem/settings
>>>>>>>> /var/ipfire/optionsfw/settings
>>>>>>>> /var/ipfire/ovpn
>>>>>>>> /var/ipfire/ovpn/ccd.conf
>>>>>>>> /var/ipfire/ovpn/collectd.vpn
>>>>>>>> /var/ipfire/ovpn/enable
>>>>>>>> /var/ipfire/ovpn/server.conf
>>>>>>>> /var/ipfire/ovpn/settings
>>>>>>>> /var/ipfire/pakfire/settings
>>>>>>>> /var/ipfire/ppp
>>>>>>>> /var/ipfire/ppp/fake-resolv.conf
>>>>>>>> /var/ipfire/ppp/settings
>>>>>>>> /var/ipfire/private/cakey.pem
>>>>>>>> /var/ipfire/proxy
>>>>>>>> /var/ipfire/proxy/asnbl-helper.conf
>>>>>>>> /var/ipfire/proxy/cachemgr.conf
>>>>>>>> /var/ipfire/proxy/enable
>>>>>>>> /var/ipfire/proxy/settings
>>>>>>>> /var/ipfire/proxy/squid.conf
>>>>>>>> /var/ipfire/qos/bin
>>>>>>>> /var/ipfire/qos/bin/qos.sh
>>>>>>>> /var/ipfire/qos/classes
>>>>>>>> /var/ipfire/qos/level7config
>>>>>>>> /var/ipfire/qos/portconfig
>>>>>>>> /var/ipfire/qos/settings
>>>>>>>> /var/ipfire/qos/subclasses
>>>>>>>> /var/ipfire/qos/tosconfig
>>>>>>>> /var/ipfire/remote/enablessh
>>>>>>>> /var/ipfire/remote/settings
>>>>>>>> /var/ipfire/sensors/settings
>>>>>>>> /var/ipfire/suricata/oinkmaster.conf
>>>>>>>> /var/ipfire/suricata/oinkmaster-modify-sids.conf
>>>>>>>> /var/ipfire/suricata/oinkmaster-provider-includes.conf
>>>>>>>> /var/ipfire/suricata/providers-settings
>>>>>>>> /var/ipfire/suricata/settings
>>>>>>>> /var/ipfire/suricata/suricata-default-rules.yaml
>>>>>>>> /var/ipfire/suricata/suricata-dns-servers.yaml
>>>>>>>> /var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
>>>>>>>> /var/ipfire/suricata/suricata-homenet.yaml
>>>>>>>> /var/ipfire/suricata/suricata-http-ports.yaml
>>>>>>>> /var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml
>>>>>>>> /var/ipfire/suricata/suricata-used-providers.yaml
>>>>>>>> /var/ipfire/time/
>>>>>>>> /var/ipfire/time/counter.conf
>>>>>>>> /var/ipfire/time/enable
>>>>>>>> /var/ipfire/time/settime.conf
>>>>>>>> /var/ipfire/time/settings
>>>>>>>> /var/ipfire/upnp/settings
>>>>>>>> /var/ipfire/urlfilter
>>>>>>>> /var/ipfire/urlfilter/settings
>>>>>>>> /var/ipfire/urlfilter/squidGuard.conf
>>>>>>>> /var/ipfire/vpn
>>>>>>>> /var/ipfire/vpn/config
>>>>>>>> /var/ipfire/vpn/ipsec.conf
>>>>>>>> /var/ipfire/vpn/settings
>>>>>>>> /var/ipfire/wakeonlan/clients.conf
>>>>>>>> /var/ipfire/wio/wio.conf
>>>>>>>> /var/ipfire/wireless/config
>>>>>>>> /var/ipfire/wireless/settings
>>>>>>>> /var/lib/suricata
>>>>>>>> /var/log/rrd/collectd
>>>>>>>> /var/log/rrd/hddshutdown-md127.rrd
>>>>>>>> /var/log/rrd/hddshutdown-sda.rrd
>>>>>>>> /var/log/rrd/hddshutdown-sdb.rrd
>>>>>>>> /var/log/rrd/hddtemp-md127.rrd
>>>>>>>> /var/log/rrd/hddtemp-sda.rrd
>>>>>>>> /var/log/rrd/hddtemp-sdb.rrd
>>>>>>>> /var/log/rrd/wio
>>>>>>>> /var/log/vnstat
>>>>>>>> /var/tmp/idsrules-emerging.tar.gz
>>>>>>>> /var/tmp/idsrules-sslbl_blacklist.rules
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Adolf.
>>>>>>>>>
>>>>>>>>> Bernhard
>>>>>>>>>> -Michael
>>>>>>>>>>
>>>>>>>>>>> On 29 Mar 2022, at 14:10, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>
>>>>>>>>>>> Just tried this patch out on my vm testbed system and it still doesn't work for me. The backup file had got to 1.3GB when I deleted the backup file as it was still growing. The normal correct backup file on that vm machine is around 7MB
>>>>>>>>>>>
>>>>>>>>>>> The above was the case for both running it from the WUI or from the command line from my unprivileged user using sudo backupctrl exclude
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>>
>>>>>>>>>>> Adolf.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 29/03/2022 14:27, Michael Tremer wrote:
>>>>>>>>>>>> This patch fixes globbing expansion in the backup include file list
>>>>>>>>>>>> which got broken in c7e0d73e7cfd7be95db9d0a5f3392b8241813d5b.
>>>>>>>>>>>>
>>>>>>>>>>>> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
>>>>>>>>>>>> ---
>>>>>>>>>>>>     config/backup/backup.pl | 8 ++++----
>>>>>>>>>>>>     1 file changed, 4 insertions(+), 4 deletions(-)
>>>>>>>>>>>>
>>>>>>>>>>>> diff --git a/config/backup/backup.pl b/config/backup/backup.pl
>>>>>>>>>>>> index a2337cf23..6f9295e94 100644
>>>>>>>>>>>> --- a/config/backup/backup.pl
>>>>>>>>>>>> +++ b/config/backup/backup.pl
>>>>>>>>>>>> @@ -19,6 +19,8 @@
>>>>>>>>>>>> # #
>>>>>>>>>>>> ###############################################################################
>>>>>>>>>>>>     +shopt -s nullglob
>>>>>>>>>>>> +
>>>>>>>>>>>>     NOW="$(date "+%Y-%m-%d-%H:%M")"
>>>>>>>>>>>>       list_addons() {
>>>>>>>>>>>> @@ -38,10 +40,8 @@ process_includes() {
>>>>>>>>>>>>         for include in $@; do
>>>>>>>>>>>>             local file
>>>>>>>>>>>>             while read -r file; do
>>>>>>>>>>>> -            for file in ${file}; do
>>>>>>>>>>>> -                if [ -e "/${file}" ]; then
>>>>>>>>>>>> -                    echo "${file}"
>>>>>>>>>>>> -                fi
>>>>>>>>>>>> +            for file in /${file}; do
>>>>>>>>>>>> +                echo "${file}"
>>>>>>>>>>>>                 done
>>>>>>>>>>>>             done < "${include}"
>>>>>>>>>>>>         done | sort -u
>>>>>>>>>>
>>>>>
>

Michael Tremer March 30, 2022, 3:05 p.m. UTC | #16

Hello Bernhard,

You are obviously right. My brain is absolute mush these days and doesn’t allow me to function.

I will correct this by the end of the day.

-Michael

> On 30 Mar 2022, at 15:38, Bernhard Bitsch <bbitsch@ipfire.org> wrote:
> 
> Hello all,
> 
> 
> Am 30.03.2022 um 16:06 schrieb Michael Tremer:
>> Hello everyone,
>> Thank you very much for helping me test this.
>> I decided that we will have to create an emergency core update for this, because it will otherwise break people’s system. We also have a vulnerability in zlib which is being handed around by the press, so that has helped me to make a decision.
> 
> I think this is a good solution. Shall we add a warning to backup wiki page? Because in the moment a backup -> reinstall -> restore process leaves an incomplete system.
> 
>> I also pushed two more patches. One filters out any empty lines because they have quite a damaging effect and it happens easily that they are added to the include list. Secondly, I brought back the existence check which I thought was only there for the globbing mechanism, but also had the small side-effect of filtering out the file list early on which helps keeping tar happy.
>> Please review those changes and please install the new c166 from testing as soon as the build has finished (I am not sure if you are all on the nightly-builds list to see any notifications).
>> https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=4f0e7f24f293ca10096e616010cfd4417afc157c
> The test on empty string should be in the 'while read -r file' loop. The empty lines are in the include files. The outer for loop iterates over the file list.
> 
> Regards,
> Bernhard
> 
>> https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=b275771fdd10ac7116e5d400dc10d3148df8ac27
>> Best,
>> -Michael
>>> On 29 Mar 2022, at 20:21, Bernhard Bitsch <bbitsch@ipfire.org> wrote:
>>> 
>>> Just checked the blank line problem.
>>> Pattern is empty , file name list is '/' --> no names produced ( with existence check )
>>> Pattern is empty --> produced file name '/' ( without -e check )
>>> 
>>> Am 29.03.2022 um 20:36 schrieb Bernhard Bitsch:
>>>> Hi all,
>>>> Am 29.03.2022 um 17:23 schrieb Adolf Belka:
>>>>> Hi All,
>>>>> 
>>>>> On 29/03/2022 16:22, Michael Tremer wrote:
>>>>>> 
>>>>>>> On 29 Mar 2022, at 15:22, Bernhard Bitsch <bbitsch@ipfire.org> wrote:
>>>>>>> 
>>>>>>> Looks like your include.user file contains absolute paths.
>>>>> That was not the problem but I have corrected that. I would have expected that when the main include and exclude files were made relative that the update script would also have checked for any entries in the include.user and exclude.user files but it looks like it didn't.th
>>>> According to the source they are chosen.
>>>>>> 
>>>>>> Or an empty line maybe?
>>>>> That turned out to be the problem. There was an empty line after the two lines I had added. You only saw it if you scrolled the pointer down the file in the editor.
>>>> Interesting. A blank line should produce nothing.
>>>>> 
>>>>> Removing that blank line removed the line only with a / and the backup then successfully ran and created a backup file containing all the directories expected.
>>>>> 
>>>>> The only thing I found is that there was the following tar error message
>>>>> 
>>>>> tar: Exiting with failure status due to previous errors
>>>>> 
>>>>> The only thing I found in the backup output was several lines such as
>>>>> 
>>>>> tar: /etc/squid/squid.conf.local: Cannot stat: No such file or directory
>>>>> tar: /etc/squid/squid.conf.pre.local: Cannot stat: No such file or directory
>>>>> 
>>>>> There were seven lines in total like this and the three I checked were files that did not exist on my system. I presume that these are files that should be backed up if present but don't have to be present so that the error messages are nothing to worry about.
>>>>> There were no error messages when doing the backup via the WUI.
>>>>> 
>>>> Good find. ;)
>>>> Michael's patch removes the test of existence, globbing produces existent file names only. But I suppose there are several full names of files, not existing in each implementation.
>>>> Conclusion: Repair the file globbing by the shopt and do the existence check. So all existent files are chosen, that are listed in the include files.
>>>> Regards,
>>>> Bernhard
>>>>> The blank line was only on my vm testbed system but as I clone this for any testing activities I will have to check all of my existing vm's to fix this.
>>>>> 
>>>>> Glad we resolved this simply. Sorry for the anxiety.
>>>>> 
>>>>> Regards,
>>>>> 
>>>>> Adolf.
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> Am 29.03.2022 um 16:14 schrieb Adolf Belka:
>>>>>>>> Hi All,
>>>>>>>> This is what I get with the pushd/popd version in place on the same system
>>>>>>>> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
>>>>>>>> etc/group
>>>>>>>> etc/hosts
>>>>>>>> etc/hosts.allow
>>>>>>>> etc/hosts.deny
>>>>>>>> etc/httpd/server.crt
>>>>>>>> etc/httpd/server.csr
>>>>>>>> etc/httpd/server-ecdsa.crt
>>>>>>>> etc/httpd/server-ecdsa.csr
>>>>>>>> etc/httpd/server-ecdsa.key
>>>>>>>> etc/httpd/server.key
>>>>>>>> etc/ipsec.user.conf
>>>>>>>> etc/ipsec.user.secrets
>>>>>>>> etc/logrotate.d
>>>>>>>> etc/passwd
>>>>>>>> etc/shadow
>>>>>>>> etc/ssh/sshd_config
>>>>>>>> etc/ssh/ssh_host_ecdsa_key
>>>>>>>> etc/ssh/ssh_host_ecdsa_key.pub
>>>>>>>> etc/ssh/ssh_host_ed25519_key
>>>>>>>> etc/ssh/ssh_host_ed25519_key.pub
>>>>>>>> etc/ssh/ssh_host_rsa_key
>>>>>>>> etc/ssh/ssh_host_rsa_key.pub
>>>>>>>> /etc/sudoers
>>>>>>>> etc/sysconfig/createfiles
>>>>>>>> etc/sysconfig/firewall.local
>>>>>>>> etc/sysconfig/lm_sensors
>>>>>>>> etc/sysconfig/modules
>>>>>>>> etc/sysconfig/ramdisk
>>>>>>>> etc/sysconfig/rc
>>>>>>>> etc/sysconfig/rc.local
>>>>>>>> etc/unbound
>>>>>>>> /home/ahb
>>>>>>>> root/.bash_history
>>>>>>>> var/ipfire/accounting/settings.conf
>>>>>>>> var/ipfire/auth/users
>>>>>>>> var/ipfire/backup/addons/backup
>>>>>>>> var/ipfire/backup/exclude.user
>>>>>>>> var/ipfire/backup/include.user
>>>>>>>> var/ipfire/ca/cacert.pem
>>>>>>>> var/ipfire/captive/agb.txt
>>>>>>>> var/ipfire/captive/clients
>>>>>>>> var/ipfire/captive/coupons
>>>>>>>> var/ipfire/captive/logo.dat
>>>>>>>> var/ipfire/captive/settings
>>>>>>>> var/ipfire/captive/terms.txt
>>>>>>>> var/ipfire/captive/voucher_out
>>>>>>>> var/ipfire/certs/hostcert.pem
>>>>>>>> var/ipfire/certs/hostkey.pem
>>>>>>>> var/ipfire/certs/phoebevmipseccert.pem
>>>>>>>> var/ipfire/connscheduler/connscheduler.conf
>>>>>>>> var/ipfire/crls/cacrl.pem
>>>>>>>> var/ipfire/cups/cups-browsed.conf
>>>>>>>> var/ipfire/cups/subscriptions.conf
>>>>>>>> var/ipfire/ddns/config
>>>>>>>> var/ipfire/ddns/ddns.conf
>>>>>>>> var/ipfire/ddns/settings
>>>>>>>> var/ipfire/dhcp/advoptions
>>>>>>>> var/ipfire/dhcp/advoptions-list
>>>>>>>> var/ipfire/dhcpc/dhcpcd.conf
>>>>>>>> var/ipfire/dhcp/dhcpd.conf
>>>>>>>> var/ipfire/dhcp/dhcpd.conf.local
>>>>>>>> var/ipfire/dhcp/enable_blue
>>>>>>>> var/ipfire/dhcp/enable_green
>>>>>>>> var/ipfire/dhcp/fixleases
>>>>>>>> var/ipfire/dhcp/settings
>>>>>>>> var/ipfire/dma/auth.conf
>>>>>>>> var/ipfire/dma/dma.conf
>>>>>>>> var/ipfire/dma/mail.conf
>>>>>>>> var/ipfire/dns
>>>>>>>> var/ipfire/dnsforward/config
>>>>>>>> var/ipfire/dns/settings
>>>>>>>> var/ipfire/ethernet/aliases
>>>>>>>> var/ipfire/ethernet/settings
>>>>>>>> var/ipfire/ethernet/wireless
>>>>>>>> var/ipfire/extrahd/settings
>>>>>>>> var/ipfire/firewall
>>>>>>>> var/ipfire/firewall/config
>>>>>>>> var/ipfire/firewall/settings
>>>>>>>> var/ipfire/fwhosts
>>>>>>>> var/ipfire/isdn/settings
>>>>>>>> var/ipfire/logging/settings
>>>>>>>> var/ipfire/mac/settings
>>>>>>>> var/ipfire/main/firstsetup_ok
>>>>>>>> var/ipfire/main/gpl_accepted
>>>>>>>> var/ipfire/main/hostname.conf
>>>>>>>> var/ipfire/main/hosts
>>>>>>>> var/ipfire/main/manualpages
>>>>>>>> var/ipfire/main/routing
>>>>>>>> var/ipfire/main/security
>>>>>>>> var/ipfire/main/send_profile
>>>>>>>> var/ipfire/main/settings
>>>>>>>> var/ipfire/modem/settings
>>>>>>>> var/ipfire/optionsfw/settings
>>>>>>>> var/ipfire/ovpn
>>>>>>>> var/ipfire/ovpn/ccd.conf
>>>>>>>> var/ipfire/ovpn/collectd.vpn
>>>>>>>> var/ipfire/ovpn/enable
>>>>>>>> var/ipfire/ovpn/server.conf
>>>>>>>> var/ipfire/ovpn/settings
>>>>>>>> var/ipfire/pakfire/settings
>>>>>>>> var/ipfire/ppp
>>>>>>>> var/ipfire/ppp/fake-resolv.conf
>>>>>>>> var/ipfire/ppp/settings
>>>>>>>> var/ipfire/private/cakey.pem
>>>>>>>> var/ipfire/proxy
>>>>>>>> var/ipfire/proxy/asnbl-helper.conf
>>>>>>>> var/ipfire/proxy/cachemgr.conf
>>>>>>>> var/ipfire/proxy/enable
>>>>>>>> var/ipfire/proxy/settings
>>>>>>>> var/ipfire/proxy/squid.conf
>>>>>>>> var/ipfire/qos/bin
>>>>>>>> var/ipfire/qos/classes
>>>>>>>> var/ipfire/qos/level7config
>>>>>>>> var/ipfire/qos/portconfig
>>>>>>>> var/ipfire/qos/settings
>>>>>>>> var/ipfire/qos/subclasses
>>>>>>>> var/ipfire/qos/tosconfig
>>>>>>>> var/ipfire/remote/enablessh
>>>>>>>> var/ipfire/remote/settings
>>>>>>>> var/ipfire/sensors/settings
>>>>>>>> var/ipfire/suricata/oinkmaster.conf
>>>>>>>> var/ipfire/suricata/oinkmaster-modify-sids.conf
>>>>>>>> var/ipfire/suricata/oinkmaster-provider-includes.conf
>>>>>>>> var/ipfire/suricata/providers-settings
>>>>>>>> var/ipfire/suricata/settings
>>>>>>>> var/ipfire/suricata/suricata-default-rules.yaml
>>>>>>>> var/ipfire/suricata/suricata-dns-servers.yaml
>>>>>>>> var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
>>>>>>>> var/ipfire/suricata/suricata-homenet.yaml
>>>>>>>> var/ipfire/suricata/suricata-http-ports.yaml
>>>>>>>> var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml
>>>>>>>> var/ipfire/suricata/suricata-used-providers.yaml
>>>>>>>> var/ipfire/time/
>>>>>>>> var/ipfire/time/counter.conf
>>>>>>>> var/ipfire/time/enable
>>>>>>>> var/ipfire/time/settime.conf
>>>>>>>> var/ipfire/time/settings
>>>>>>>> var/ipfire/upnp/settings
>>>>>>>> var/ipfire/urlfilter
>>>>>>>> var/ipfire/urlfilter/settings
>>>>>>>> var/ipfire/urlfilter/squidGuard.conf
>>>>>>>> var/ipfire/vpn
>>>>>>>> var/ipfire/vpn/config
>>>>>>>> var/ipfire/vpn/ipsec.conf
>>>>>>>> var/ipfire/vpn/settings
>>>>>>>> var/ipfire/wakeonlan/clients.conf
>>>>>>>> var/ipfire/wio/wio.conf
>>>>>>>> var/ipfire/wireless/config
>>>>>>>> var/ipfire/wireless/settings
>>>>>>>> var/lib/suricata
>>>>>>>> var/log/rrd/collectd
>>>>>>>> var/log/rrd/hddshutdown-md127.rrd
>>>>>>>> var/log/rrd/hddshutdown-sda.rrd
>>>>>>>> var/log/rrd/hddshutdown-sdb.rrd
>>>>>>>> var/log/rrd/hddtemp-md127.rrd
>>>>>>>> var/log/rrd/hddtemp-sda.rrd
>>>>>>>> var/log/rrd/hddtemp-sdb.rrd
>>>>>>>> var/log/rrd/wio
>>>>>>>> var/log/vnstat
>>>>>>>> var/tmp/idsrules-emerging.tar.gz
>>>>>>>> var/tmp/idsrules-sslbl_blacklist.rules
>>>>>>>> The following are in the previous list but not in this one:-
>>>>>>>> /
>>>>>>>> /etc/conntrackd/conntrackd.conf
>>>>>>>> /etc/ipsec.user-post.conf
>>>>>>>> /root/.gitconfig
>>>>>>>> /root/.ssh
>>>>>>>> Regards,
>>>>>>>> Adolf.
>>>>>>>> On 29/03/2022 16:01, Adolf Belka wrote:
>>>>>>>>> Hi All,
>>>>>>>>> 
>>>>>>>>> On 29/03/2022 15:36, Bernhard Bitsch wrote:
>>>>>>>>>> Hi,
>>>>>>>>>> 
>>>>>>>>>> Am 29.03.2022 um 15:11 schrieb Michael Tremer:
>>>>>>>>>>> Hello,
>>>>>>>>>>> 
>>>>>>>>>>> Could you please send the file listing to find out what is being included what shouldn’t?
>>>>>>>>> Unfortunately, as I stopped the backup continuing once it had reached 1.2GB, the file created was not able to be opened. Probably stopping the backup corrupted it in some way.
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Just for easy quick handling, you can use
>>>>>>>>>> '/var/ipfire/backup/bin/backup.pl list' to show the files included.
>>>>>>>>> I gave this a go and here is the output from the command.
>>>>>>>>> 
>>>>>>>>> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
>>>>>>>>> /
>>>>>>>>> /etc/conntrackd/conntrackd.conf
>>>>>>>>> /etc/group
>>>>>>>>> /etc/hosts
>>>>>>>>> /etc/hosts.allow
>>>>>>>>> /etc/hosts.deny
>>>>>>>>> /etc/httpd/server.crt
>>>>>>>>> /etc/httpd/server.csr
>>>>>>>>> /etc/httpd/server-ecdsa.crt
>>>>>>>>> /etc/httpd/server-ecdsa.csr
>>>>>>>>> /etc/httpd/server-ecdsa.key
>>>>>>>>> /etc/httpd/server.key
>>>>>>>>> /etc/ipsec.user.conf
>>>>>>>>> /etc/ipsec.user-post.conf
>>>>>>>>> /etc/ipsec.user.secrets
>>>>>>>>> /etc/logrotate.d
>>>>>>>>> /etc/passwd
>>>>>>>>> /etc/shadow
>>>>>>>>> /etc/squid/squid.conf.local
>>>>>>>>> /etc/squid/squid.conf.pre.local
>>>>>>>>> /etc/ssh/sshd_config
>>>>>>>>> /etc/ssh/ssh_host_ecdsa_key
>>>>>>>>> /etc/ssh/ssh_host_ecdsa_key.pub
>>>>>>>>> /etc/ssh/ssh_host_ed25519_key
>>>>>>>>> /etc/ssh/ssh_host_ed25519_key.pub
>>>>>>>>> /etc/ssh/ssh_host_rsa_key
>>>>>>>>> /etc/ssh/ssh_host_rsa_key.pub
>>>>>>>>> //etc/sudoers
>>>>>>>>> /etc/sysconfig/createfiles
>>>>>>>>> /etc/sysconfig/firewall.local
>>>>>>>>> /etc/sysconfig/lm_sensors
>>>>>>>>> /etc/sysconfig/modules
>>>>>>>>> /etc/sysconfig/ramdisk
>>>>>>>>> /etc/sysconfig/rc
>>>>>>>>> /etc/sysconfig/rc.local
>>>>>>>>> /etc/unbound
>>>>>>>>> //home/ahb
>>>>>>>>> /root/.bash_history
>>>>>>>>> /root/.gitconfig
>>>>>>>>> /root/.ssh
>>>>>>>>> /var/ipfire/accounting/settings.conf
>>>>>>>>> /var/ipfire/auth/users
>>>>>>>>> /var/ipfire/backup/addons/backup
>>>>>>>>> /var/ipfire/backup/exclude.user
>>>>>>>>> /var/ipfire/backup/include.user
>>>>>>>>> /var/ipfire/ca/cacert.pem
>>>>>>>>> /var/ipfire/captive/agb.txt
>>>>>>>>> /var/ipfire/captive/clients
>>>>>>>>> /var/ipfire/captive/coupons
>>>>>>>>> /var/ipfire/captive/logo.dat
>>>>>>>>> /var/ipfire/captive/settings
>>>>>>>>> /var/ipfire/captive/terms.txt
>>>>>>>>> /var/ipfire/captive/voucher_out
>>>>>>>>> /var/ipfire/certs/hostcert.pem
>>>>>>>>> /var/ipfire/certs/hostkey.pem
>>>>>>>>> /var/ipfire/certs/phoebevmipseccert.pem
>>>>>>>>> /var/ipfire/connscheduler/connscheduler.conf
>>>>>>>>> /var/ipfire/crls/cacrl.pem
>>>>>>>>> /var/ipfire/cups/cups-browsed.conf
>>>>>>>>> /var/ipfire/cups/subscriptions.conf
>>>>>>>>> /var/ipfire/ddns/config
>>>>>>>>> /var/ipfire/ddns/ddns.conf
>>>>>>>>> /var/ipfire/ddns/settings
>>>>>>>>> /var/ipfire/dhcp/advoptions
>>>>>>>>> /var/ipfire/dhcp/advoptions-list
>>>>>>>>> /var/ipfire/dhcpc/dhcpcd.conf
>>>>>>>>> /var/ipfire/dhcp/dhcpd.conf
>>>>>>>>> /var/ipfire/dhcp/dhcpd.conf.local
>>>>>>>>> /var/ipfire/dhcp/enable_blue
>>>>>>>>> /var/ipfire/dhcp/enable_green
>>>>>>>>> /var/ipfire/dhcp/fixleases
>>>>>>>>> /var/ipfire/dhcp/settings
>>>>>>>>> /var/ipfire/dma/auth.conf
>>>>>>>>> /var/ipfire/dma/dma.conf
>>>>>>>>> /var/ipfire/dma/mail.conf
>>>>>>>>> /var/ipfire/dns
>>>>>>>>> /var/ipfire/dnsforward/config
>>>>>>>>> /var/ipfire/dns/settings
>>>>>>>>> /var/ipfire/ethernet/aliases
>>>>>>>>> /var/ipfire/ethernet/settings
>>>>>>>>> /var/ipfire/ethernet/wireless
>>>>>>>>> /var/ipfire/extrahd/settings
>>>>>>>>> /var/ipfire/firewall
>>>>>>>>> /var/ipfire/firewall/config
>>>>>>>>> /var/ipfire/firewall/settings
>>>>>>>>> /var/ipfire/fwhosts
>>>>>>>>> /var/ipfire/isdn/settings
>>>>>>>>> /var/ipfire/logging/settings
>>>>>>>>> /var/ipfire/mac/settings
>>>>>>>>> /var/ipfire/main/firstsetup_ok
>>>>>>>>> /var/ipfire/main/gpl_accepted
>>>>>>>>> /var/ipfire/main/hostname.conf
>>>>>>>>> /var/ipfire/main/hosts
>>>>>>>>> /var/ipfire/main/manualpages
>>>>>>>>> /var/ipfire/main/routing
>>>>>>>>> /var/ipfire/main/security
>>>>>>>>> /var/ipfire/main/send_profile
>>>>>>>>> /var/ipfire/main/settings
>>>>>>>>> /var/ipfire/modem/settings
>>>>>>>>> /var/ipfire/optionsfw/settings
>>>>>>>>> /var/ipfire/ovpn
>>>>>>>>> /var/ipfire/ovpn/ccd.conf
>>>>>>>>> /var/ipfire/ovpn/collectd.vpn
>>>>>>>>> /var/ipfire/ovpn/enable
>>>>>>>>> /var/ipfire/ovpn/server.conf
>>>>>>>>> /var/ipfire/ovpn/settings
>>>>>>>>> /var/ipfire/pakfire/settings
>>>>>>>>> /var/ipfire/ppp
>>>>>>>>> /var/ipfire/ppp/fake-resolv.conf
>>>>>>>>> /var/ipfire/ppp/settings
>>>>>>>>> /var/ipfire/private/cakey.pem
>>>>>>>>> /var/ipfire/proxy
>>>>>>>>> /var/ipfire/proxy/asnbl-helper.conf
>>>>>>>>> /var/ipfire/proxy/cachemgr.conf
>>>>>>>>> /var/ipfire/proxy/enable
>>>>>>>>> /var/ipfire/proxy/settings
>>>>>>>>> /var/ipfire/proxy/squid.conf
>>>>>>>>> /var/ipfire/qos/bin
>>>>>>>>> /var/ipfire/qos/bin/qos.sh
>>>>>>>>> /var/ipfire/qos/classes
>>>>>>>>> /var/ipfire/qos/level7config
>>>>>>>>> /var/ipfire/qos/portconfig
>>>>>>>>> /var/ipfire/qos/settings
>>>>>>>>> /var/ipfire/qos/subclasses
>>>>>>>>> /var/ipfire/qos/tosconfig
>>>>>>>>> /var/ipfire/remote/enablessh
>>>>>>>>> /var/ipfire/remote/settings
>>>>>>>>> /var/ipfire/sensors/settings
>>>>>>>>> /var/ipfire/suricata/oinkmaster.conf
>>>>>>>>> /var/ipfire/suricata/oinkmaster-modify-sids.conf
>>>>>>>>> /var/ipfire/suricata/oinkmaster-provider-includes.conf
>>>>>>>>> /var/ipfire/suricata/providers-settings
>>>>>>>>> /var/ipfire/suricata/settings
>>>>>>>>> /var/ipfire/suricata/suricata-default-rules.yaml
>>>>>>>>> /var/ipfire/suricata/suricata-dns-servers.yaml
>>>>>>>>> /var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
>>>>>>>>> /var/ipfire/suricata/suricata-homenet.yaml
>>>>>>>>> /var/ipfire/suricata/suricata-http-ports.yaml
>>>>>>>>> /var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml
>>>>>>>>> /var/ipfire/suricata/suricata-used-providers.yaml
>>>>>>>>> /var/ipfire/time/
>>>>>>>>> /var/ipfire/time/counter.conf
>>>>>>>>> /var/ipfire/time/enable
>>>>>>>>> /var/ipfire/time/settime.conf
>>>>>>>>> /var/ipfire/time/settings
>>>>>>>>> /var/ipfire/upnp/settings
>>>>>>>>> /var/ipfire/urlfilter
>>>>>>>>> /var/ipfire/urlfilter/settings
>>>>>>>>> /var/ipfire/urlfilter/squidGuard.conf
>>>>>>>>> /var/ipfire/vpn
>>>>>>>>> /var/ipfire/vpn/config
>>>>>>>>> /var/ipfire/vpn/ipsec.conf
>>>>>>>>> /var/ipfire/vpn/settings
>>>>>>>>> /var/ipfire/wakeonlan/clients.conf
>>>>>>>>> /var/ipfire/wio/wio.conf
>>>>>>>>> /var/ipfire/wireless/config
>>>>>>>>> /var/ipfire/wireless/settings
>>>>>>>>> /var/lib/suricata
>>>>>>>>> /var/log/rrd/collectd
>>>>>>>>> /var/log/rrd/hddshutdown-md127.rrd
>>>>>>>>> /var/log/rrd/hddshutdown-sda.rrd
>>>>>>>>> /var/log/rrd/hddshutdown-sdb.rrd
>>>>>>>>> /var/log/rrd/hddtemp-md127.rrd
>>>>>>>>> /var/log/rrd/hddtemp-sda.rrd
>>>>>>>>> /var/log/rrd/hddtemp-sdb.rrd
>>>>>>>>> /var/log/rrd/wio
>>>>>>>>> /var/log/vnstat
>>>>>>>>> /var/tmp/idsrules-emerging.tar.gz
>>>>>>>>> /var/tmp/idsrules-sslbl_blacklist.rules
>>>>>>>>> 
>>>>>>>>> Regards,
>>>>>>>>> Adolf.
>>>>>>>>>> 
>>>>>>>>>> Bernhard
>>>>>>>>>>> -Michael
>>>>>>>>>>> 
>>>>>>>>>>>> On 29 Mar 2022, at 14:10, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>> 
>>>>>>>>>>>> Just tried this patch out on my vm testbed system and it still doesn't work for me. The backup file had got to 1.3GB when I deleted the backup file as it was still growing. The normal correct backup file on that vm machine is around 7MB
>>>>>>>>>>>> 
>>>>>>>>>>>> The above was the case for both running it from the WUI or from the command line from my unprivileged user using sudo backupctrl exclude
>>>>>>>>>>>> 
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> 
>>>>>>>>>>>> Adolf.
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> On 29/03/2022 14:27, Michael Tremer wrote:
>>>>>>>>>>>>> This patch fixes globbing expansion in the backup include file list
>>>>>>>>>>>>> which got broken in c7e0d73e7cfd7be95db9d0a5f3392b8241813d5b.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
>>>>>>>>>>>>> ---
>>>>>>>>>>>>>    config/backup/backup.pl | 8 ++++----
>>>>>>>>>>>>>    1 file changed, 4 insertions(+), 4 deletions(-)
>>>>>>>>>>>>> 
>>>>>>>>>>>>> diff --git a/config/backup/backup.pl b/config/backup/backup.pl
>>>>>>>>>>>>> index a2337cf23..6f9295e94 100644
>>>>>>>>>>>>> --- a/config/backup/backup.pl
>>>>>>>>>>>>> +++ b/config/backup/backup.pl
>>>>>>>>>>>>> @@ -19,6 +19,8 @@
>>>>>>>>>>>>> # #
>>>>>>>>>>>>> ###############################################################################
>>>>>>>>>>>>>    +shopt -s nullglob
>>>>>>>>>>>>> +
>>>>>>>>>>>>>    NOW="$(date "+%Y-%m-%d-%H:%M")"
>>>>>>>>>>>>>      list_addons() {
>>>>>>>>>>>>> @@ -38,10 +40,8 @@ process_includes() {
>>>>>>>>>>>>>        for include in $@; do
>>>>>>>>>>>>>            local file
>>>>>>>>>>>>>            while read -r file; do
>>>>>>>>>>>>> -            for file in ${file}; do
>>>>>>>>>>>>> -                if [ -e "/${file}" ]; then
>>>>>>>>>>>>> -                    echo "${file}"
>>>>>>>>>>>>> -                fi
>>>>>>>>>>>>> +            for file in /${file}; do
>>>>>>>>>>>>> +                echo "${file}"
>>>>>>>>>>>>>                done
>>>>>>>>>>>>>            done < "${include}"
>>>>>>>>>>>>>        done | sort -u
>>>>>>>>>>> 
>>>>>>

Jon Murphy March 30, 2022, 7:47 p.m. UTC | #17

Hello *.*,


> On Mar 30, 2022, at 9:38 AM, Bernhard Bitsch <bbitsch@ipfire.org> wrote:
> 
> Hello all,
> 
> 
> Am 30.03.2022 um 16:06 schrieb Michael Tremer:
>> Hello everyone,
>> Thank you very much for helping me test this.
>> I decided that we will have to create an emergency core update for this, because it will otherwise break people’s system. We also have a vulnerability in zlib which is being handed around by the press, so that has helped me to make a decision.
> 
> I think this is a good solution. Shall we add a warning to backup wiki page? Because in the moment a backup -> reinstall -> restore process leaves an incomplete system.


I added a "pinned" note on the IPFire Community.  
https://community.ipfire.org/t/cu-164-backup-issue/7614 <https://community.ipfire.org/t/cu-164-backup-issue/7614>

If this is incorrect, please feel free to make changes as needed.  Most of you have moderator privileges. 


Jon


> 
>> I also pushed two more patches. One filters out any empty lines because they have quite a damaging effect and it happens easily that they are added to the include list. Secondly, I brought back the existence check which I thought was only there for the globbing mechanism, but also had the small side-effect of filtering out the file list early on which helps keeping tar happy.
>> Please review those changes and please install the new c166 from testing as soon as the build has finished (I am not sure if you are all on the nightly-builds list to see any notifications).
>> https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=4f0e7f24f293ca10096e616010cfd4417afc157c
> The test on empty string should be in the 'while read -r file' loop. The empty lines are in the include files. The outer for loop iterates over the file list.
> 
> Regards,
> Bernhard
> 
>> https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=b275771fdd10ac7116e5d400dc10d3148df8ac27
>> Best,
>> -Michael
>>> On 29 Mar 2022, at 20:21, Bernhard Bitsch <bbitsch@ipfire.org> wrote:
>>> 
>>> Just checked the blank line problem.
>>> Pattern is empty , file name list is '/' --> no names produced ( with existence check )
>>> Pattern is empty --> produced file name '/' ( without -e check )
>>> 
>>> Am 29.03.2022 um 20:36 schrieb Bernhard Bitsch:
>>>> Hi all,
>>>> Am 29.03.2022 um 17:23 schrieb Adolf Belka:
>>>>> Hi All,
>>>>> 
>>>>> On 29/03/2022 16:22, Michael Tremer wrote:
>>>>>> 
>>>>>>> On 29 Mar 2022, at 15:22, Bernhard Bitsch <bbitsch@ipfire.org> wrote:
>>>>>>> 
>>>>>>> Looks like your include.user file contains absolute paths.
>>>>> That was not the problem but I have corrected that. I would have expected that when the main include and exclude files were made relative that the update script would also have checked for any entries in the include.user and exclude.user files but it looks like it didn't.th
>>>> According to the source they are chosen.
>>>>>> 
>>>>>> Or an empty line maybe?
>>>>> That turned out to be the problem. There was an empty line after the two lines I had added. You only saw it if you scrolled the pointer down the file in the editor.
>>>> Interesting. A blank line should produce nothing.
>>>>> 
>>>>> Removing that blank line removed the line only with a / and the backup then successfully ran and created a backup file containing all the directories expected.
>>>>> 
>>>>> The only thing I found is that there was the following tar error message
>>>>> 
>>>>> tar: Exiting with failure status due to previous errors
>>>>> 
>>>>> The only thing I found in the backup output was several lines such as
>>>>> 
>>>>> tar: /etc/squid/squid.conf.local: Cannot stat: No such file or directory
>>>>> tar: /etc/squid/squid.conf.pre.local: Cannot stat: No such file or directory
>>>>> 
>>>>> There were seven lines in total like this and the three I checked were files that did not exist on my system. I presume that these are files that should be backed up if present but don't have to be present so that the error messages are nothing to worry about.
>>>>> There were no error messages when doing the backup via the WUI.
>>>>> 
>>>> Good find. ;)
>>>> Michael's patch removes the test of existence, globbing produces existent file names only. But I suppose there are several full names of files, not existing in each implementation.
>>>> Conclusion: Repair the file globbing by the shopt and do the existence check. So all existent files are chosen, that are listed in the include files.
>>>> Regards,
>>>> Bernhard
>>>>> The blank line was only on my vm testbed system but as I clone this for any testing activities I will have to check all of my existing vm's to fix this.
>>>>> 
>>>>> Glad we resolved this simply. Sorry for the anxiety.
>>>>> 
>>>>> Regards,
>>>>> 
>>>>> Adolf.
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> Am 29.03.2022 um 16:14 schrieb Adolf Belka:
>>>>>>>> Hi All,
>>>>>>>> This is what I get with the pushd/popd version in place on the same system
>>>>>>>> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
>>>>>>>> etc/group
>>>>>>>> etc/hosts
>>>>>>>> etc/hosts.allow
>>>>>>>> etc/hosts.deny
>>>>>>>> etc/httpd/server.crt
>>>>>>>> etc/httpd/server.csr
>>>>>>>> etc/httpd/server-ecdsa.crt
>>>>>>>> etc/httpd/server-ecdsa.csr
>>>>>>>> etc/httpd/server-ecdsa.key
>>>>>>>> etc/httpd/server.key
>>>>>>>> etc/ipsec.user.conf
>>>>>>>> etc/ipsec.user.secrets
>>>>>>>> etc/logrotate.d
>>>>>>>> etc/passwd
>>>>>>>> etc/shadow
>>>>>>>> etc/ssh/sshd_config
>>>>>>>> etc/ssh/ssh_host_ecdsa_key
>>>>>>>> etc/ssh/ssh_host_ecdsa_key.pub
>>>>>>>> etc/ssh/ssh_host_ed25519_key
>>>>>>>> etc/ssh/ssh_host_ed25519_key.pub
>>>>>>>> etc/ssh/ssh_host_rsa_key
>>>>>>>> etc/ssh/ssh_host_rsa_key.pub
>>>>>>>> /etc/sudoers
>>>>>>>> etc/sysconfig/createfiles
>>>>>>>> etc/sysconfig/firewall.local
>>>>>>>> etc/sysconfig/lm_sensors
>>>>>>>> etc/sysconfig/modules
>>>>>>>> etc/sysconfig/ramdisk
>>>>>>>> etc/sysconfig/rc
>>>>>>>> etc/sysconfig/rc.local
>>>>>>>> etc/unbound
>>>>>>>> /home/ahb
>>>>>>>> root/.bash_history
>>>>>>>> var/ipfire/accounting/settings.conf
>>>>>>>> var/ipfire/auth/users
>>>>>>>> var/ipfire/backup/addons/backup
>>>>>>>> var/ipfire/backup/exclude.user
>>>>>>>> var/ipfire/backup/include.user
>>>>>>>> var/ipfire/ca/cacert.pem
>>>>>>>> var/ipfire/captive/agb.txt
>>>>>>>> var/ipfire/captive/clients
>>>>>>>> var/ipfire/captive/coupons
>>>>>>>> var/ipfire/captive/logo.dat
>>>>>>>> var/ipfire/captive/settings
>>>>>>>> var/ipfire/captive/terms.txt
>>>>>>>> var/ipfire/captive/voucher_out
>>>>>>>> var/ipfire/certs/hostcert.pem
>>>>>>>> var/ipfire/certs/hostkey.pem
>>>>>>>> var/ipfire/certs/phoebevmipseccert.pem
>>>>>>>> var/ipfire/connscheduler/connscheduler.conf
>>>>>>>> var/ipfire/crls/cacrl.pem
>>>>>>>> var/ipfire/cups/cups-browsed.conf
>>>>>>>> var/ipfire/cups/subscriptions.conf
>>>>>>>> var/ipfire/ddns/config
>>>>>>>> var/ipfire/ddns/ddns.conf
>>>>>>>> var/ipfire/ddns/settings
>>>>>>>> var/ipfire/dhcp/advoptions
>>>>>>>> var/ipfire/dhcp/advoptions-list
>>>>>>>> var/ipfire/dhcpc/dhcpcd.conf
>>>>>>>> var/ipfire/dhcp/dhcpd.conf
>>>>>>>> var/ipfire/dhcp/dhcpd.conf.local
>>>>>>>> var/ipfire/dhcp/enable_blue
>>>>>>>> var/ipfire/dhcp/enable_green
>>>>>>>> var/ipfire/dhcp/fixleases
>>>>>>>> var/ipfire/dhcp/settings
>>>>>>>> var/ipfire/dma/auth.conf
>>>>>>>> var/ipfire/dma/dma.conf
>>>>>>>> var/ipfire/dma/mail.conf
>>>>>>>> var/ipfire/dns
>>>>>>>> var/ipfire/dnsforward/config
>>>>>>>> var/ipfire/dns/settings
>>>>>>>> var/ipfire/ethernet/aliases
>>>>>>>> var/ipfire/ethernet/settings
>>>>>>>> var/ipfire/ethernet/wireless
>>>>>>>> var/ipfire/extrahd/settings
>>>>>>>> var/ipfire/firewall
>>>>>>>> var/ipfire/firewall/config
>>>>>>>> var/ipfire/firewall/settings
>>>>>>>> var/ipfire/fwhosts
>>>>>>>> var/ipfire/isdn/settings
>>>>>>>> var/ipfire/logging/settings
>>>>>>>> var/ipfire/mac/settings
>>>>>>>> var/ipfire/main/firstsetup_ok
>>>>>>>> var/ipfire/main/gpl_accepted
>>>>>>>> var/ipfire/main/hostname.conf
>>>>>>>> var/ipfire/main/hosts
>>>>>>>> var/ipfire/main/manualpages
>>>>>>>> var/ipfire/main/routing
>>>>>>>> var/ipfire/main/security
>>>>>>>> var/ipfire/main/send_profile
>>>>>>>> var/ipfire/main/settings
>>>>>>>> var/ipfire/modem/settings
>>>>>>>> var/ipfire/optionsfw/settings
>>>>>>>> var/ipfire/ovpn
>>>>>>>> var/ipfire/ovpn/ccd.conf
>>>>>>>> var/ipfire/ovpn/collectd.vpn
>>>>>>>> var/ipfire/ovpn/enable
>>>>>>>> var/ipfire/ovpn/server.conf
>>>>>>>> var/ipfire/ovpn/settings
>>>>>>>> var/ipfire/pakfire/settings
>>>>>>>> var/ipfire/ppp
>>>>>>>> var/ipfire/ppp/fake-resolv.conf
>>>>>>>> var/ipfire/ppp/settings
>>>>>>>> var/ipfire/private/cakey.pem
>>>>>>>> var/ipfire/proxy
>>>>>>>> var/ipfire/proxy/asnbl-helper.conf
>>>>>>>> var/ipfire/proxy/cachemgr.conf
>>>>>>>> var/ipfire/proxy/enable
>>>>>>>> var/ipfire/proxy/settings
>>>>>>>> var/ipfire/proxy/squid.conf
>>>>>>>> var/ipfire/qos/bin
>>>>>>>> var/ipfire/qos/classes
>>>>>>>> var/ipfire/qos/level7config
>>>>>>>> var/ipfire/qos/portconfig
>>>>>>>> var/ipfire/qos/settings
>>>>>>>> var/ipfire/qos/subclasses
>>>>>>>> var/ipfire/qos/tosconfig
>>>>>>>> var/ipfire/remote/enablessh
>>>>>>>> var/ipfire/remote/settings
>>>>>>>> var/ipfire/sensors/settings
>>>>>>>> var/ipfire/suricata/oinkmaster.conf
>>>>>>>> var/ipfire/suricata/oinkmaster-modify-sids.conf
>>>>>>>> var/ipfire/suricata/oinkmaster-provider-includes.conf
>>>>>>>> var/ipfire/suricata/providers-settings
>>>>>>>> var/ipfire/suricata/settings
>>>>>>>> var/ipfire/suricata/suricata-default-rules.yaml
>>>>>>>> var/ipfire/suricata/suricata-dns-servers.yaml
>>>>>>>> var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
>>>>>>>> var/ipfire/suricata/suricata-homenet.yaml
>>>>>>>> var/ipfire/suricata/suricata-http-ports.yaml
>>>>>>>> var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml
>>>>>>>> var/ipfire/suricata/suricata-used-providers.yaml
>>>>>>>> var/ipfire/time/
>>>>>>>> var/ipfire/time/counter.conf
>>>>>>>> var/ipfire/time/enable
>>>>>>>> var/ipfire/time/settime.conf
>>>>>>>> var/ipfire/time/settings
>>>>>>>> var/ipfire/upnp/settings
>>>>>>>> var/ipfire/urlfilter
>>>>>>>> var/ipfire/urlfilter/settings
>>>>>>>> var/ipfire/urlfilter/squidGuard.conf
>>>>>>>> var/ipfire/vpn
>>>>>>>> var/ipfire/vpn/config
>>>>>>>> var/ipfire/vpn/ipsec.conf
>>>>>>>> var/ipfire/vpn/settings
>>>>>>>> var/ipfire/wakeonlan/clients.conf
>>>>>>>> var/ipfire/wio/wio.conf
>>>>>>>> var/ipfire/wireless/config
>>>>>>>> var/ipfire/wireless/settings
>>>>>>>> var/lib/suricata
>>>>>>>> var/log/rrd/collectd
>>>>>>>> var/log/rrd/hddshutdown-md127.rrd
>>>>>>>> var/log/rrd/hddshutdown-sda.rrd
>>>>>>>> var/log/rrd/hddshutdown-sdb.rrd
>>>>>>>> var/log/rrd/hddtemp-md127.rrd
>>>>>>>> var/log/rrd/hddtemp-sda.rrd
>>>>>>>> var/log/rrd/hddtemp-sdb.rrd
>>>>>>>> var/log/rrd/wio
>>>>>>>> var/log/vnstat
>>>>>>>> var/tmp/idsrules-emerging.tar.gz
>>>>>>>> var/tmp/idsrules-sslbl_blacklist.rules
>>>>>>>> The following are in the previous list but not in this one:-
>>>>>>>> /
>>>>>>>> /etc/conntrackd/conntrackd.conf
>>>>>>>> /etc/ipsec.user-post.conf
>>>>>>>> /root/.gitconfig
>>>>>>>> /root/.ssh
>>>>>>>> Regards,
>>>>>>>> Adolf.
>>>>>>>> On 29/03/2022 16:01, Adolf Belka wrote:
>>>>>>>>> Hi All,
>>>>>>>>> 
>>>>>>>>> On 29/03/2022 15:36, Bernhard Bitsch wrote:
>>>>>>>>>> Hi,
>>>>>>>>>> 
>>>>>>>>>> Am 29.03.2022 um 15:11 schrieb Michael Tremer:
>>>>>>>>>>> Hello,
>>>>>>>>>>> 
>>>>>>>>>>> Could you please send the file listing to find out what is being included what shouldn’t?
>>>>>>>>> Unfortunately, as I stopped the backup continuing once it had reached 1.2GB, the file created was not able to be opened. Probably stopping the backup corrupted it in some way.
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Just for easy quick handling, you can use
>>>>>>>>>> '/var/ipfire/backup/bin/backup.pl list' to show the files included.
>>>>>>>>> I gave this a go and here is the output from the command.
>>>>>>>>> 
>>>>>>>>> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
>>>>>>>>> /
>>>>>>>>> /etc/conntrackd/conntrackd.conf
>>>>>>>>> /etc/group
>>>>>>>>> /etc/hosts
>>>>>>>>> /etc/hosts.allow
>>>>>>>>> /etc/hosts.deny
>>>>>>>>> /etc/httpd/server.crt
>>>>>>>>> /etc/httpd/server.csr
>>>>>>>>> /etc/httpd/server-ecdsa.crt
>>>>>>>>> /etc/httpd/server-ecdsa.csr
>>>>>>>>> /etc/httpd/server-ecdsa.key
>>>>>>>>> /etc/httpd/server.key
>>>>>>>>> /etc/ipsec.user.conf
>>>>>>>>> /etc/ipsec.user-post.conf
>>>>>>>>> /etc/ipsec.user.secrets
>>>>>>>>> /etc/logrotate.d
>>>>>>>>> /etc/passwd
>>>>>>>>> /etc/shadow
>>>>>>>>> /etc/squid/squid.conf.local
>>>>>>>>> /etc/squid/squid.conf.pre.local
>>>>>>>>> /etc/ssh/sshd_config
>>>>>>>>> /etc/ssh/ssh_host_ecdsa_key
>>>>>>>>> /etc/ssh/ssh_host_ecdsa_key.pub
>>>>>>>>> /etc/ssh/ssh_host_ed25519_key
>>>>>>>>> /etc/ssh/ssh_host_ed25519_key.pub
>>>>>>>>> /etc/ssh/ssh_host_rsa_key
>>>>>>>>> /etc/ssh/ssh_host_rsa_key.pub
>>>>>>>>> //etc/sudoers
>>>>>>>>> /etc/sysconfig/createfiles
>>>>>>>>> /etc/sysconfig/firewall.local
>>>>>>>>> /etc/sysconfig/lm_sensors
>>>>>>>>> /etc/sysconfig/modules
>>>>>>>>> /etc/sysconfig/ramdisk
>>>>>>>>> /etc/sysconfig/rc
>>>>>>>>> /etc/sysconfig/rc.local
>>>>>>>>> /etc/unbound
>>>>>>>>> //home/ahb
>>>>>>>>> /root/.bash_history
>>>>>>>>> /root/.gitconfig
>>>>>>>>> /root/.ssh
>>>>>>>>> /var/ipfire/accounting/settings.conf
>>>>>>>>> /var/ipfire/auth/users
>>>>>>>>> /var/ipfire/backup/addons/backup
>>>>>>>>> /var/ipfire/backup/exclude.user
>>>>>>>>> /var/ipfire/backup/include.user
>>>>>>>>> /var/ipfire/ca/cacert.pem
>>>>>>>>> /var/ipfire/captive/agb.txt
>>>>>>>>> /var/ipfire/captive/clients
>>>>>>>>> /var/ipfire/captive/coupons
>>>>>>>>> /var/ipfire/captive/logo.dat
>>>>>>>>> /var/ipfire/captive/settings
>>>>>>>>> /var/ipfire/captive/terms.txt
>>>>>>>>> /var/ipfire/captive/voucher_out
>>>>>>>>> /var/ipfire/certs/hostcert.pem
>>>>>>>>> /var/ipfire/certs/hostkey.pem
>>>>>>>>> /var/ipfire/certs/phoebevmipseccert.pem
>>>>>>>>> /var/ipfire/connscheduler/connscheduler.conf
>>>>>>>>> /var/ipfire/crls/cacrl.pem
>>>>>>>>> /var/ipfire/cups/cups-browsed.conf
>>>>>>>>> /var/ipfire/cups/subscriptions.conf
>>>>>>>>> /var/ipfire/ddns/config
>>>>>>>>> /var/ipfire/ddns/ddns.conf
>>>>>>>>> /var/ipfire/ddns/settings
>>>>>>>>> /var/ipfire/dhcp/advoptions
>>>>>>>>> /var/ipfire/dhcp/advoptions-list
>>>>>>>>> /var/ipfire/dhcpc/dhcpcd.conf
>>>>>>>>> /var/ipfire/dhcp/dhcpd.conf
>>>>>>>>> /var/ipfire/dhcp/dhcpd.conf.local
>>>>>>>>> /var/ipfire/dhcp/enable_blue
>>>>>>>>> /var/ipfire/dhcp/enable_green
>>>>>>>>> /var/ipfire/dhcp/fixleases
>>>>>>>>> /var/ipfire/dhcp/settings
>>>>>>>>> /var/ipfire/dma/auth.conf
>>>>>>>>> /var/ipfire/dma/dma.conf
>>>>>>>>> /var/ipfire/dma/mail.conf
>>>>>>>>> /var/ipfire/dns
>>>>>>>>> /var/ipfire/dnsforward/config
>>>>>>>>> /var/ipfire/dns/settings
>>>>>>>>> /var/ipfire/ethernet/aliases
>>>>>>>>> /var/ipfire/ethernet/settings
>>>>>>>>> /var/ipfire/ethernet/wireless
>>>>>>>>> /var/ipfire/extrahd/settings
>>>>>>>>> /var/ipfire/firewall
>>>>>>>>> /var/ipfire/firewall/config
>>>>>>>>> /var/ipfire/firewall/settings
>>>>>>>>> /var/ipfire/fwhosts
>>>>>>>>> /var/ipfire/isdn/settings
>>>>>>>>> /var/ipfire/logging/settings
>>>>>>>>> /var/ipfire/mac/settings
>>>>>>>>> /var/ipfire/main/firstsetup_ok
>>>>>>>>> /var/ipfire/main/gpl_accepted
>>>>>>>>> /var/ipfire/main/hostname.conf
>>>>>>>>> /var/ipfire/main/hosts
>>>>>>>>> /var/ipfire/main/manualpages
>>>>>>>>> /var/ipfire/main/routing
>>>>>>>>> /var/ipfire/main/security
>>>>>>>>> /var/ipfire/main/send_profile
>>>>>>>>> /var/ipfire/main/settings
>>>>>>>>> /var/ipfire/modem/settings
>>>>>>>>> /var/ipfire/optionsfw/settings
>>>>>>>>> /var/ipfire/ovpn
>>>>>>>>> /var/ipfire/ovpn/ccd.conf
>>>>>>>>> /var/ipfire/ovpn/collectd.vpn
>>>>>>>>> /var/ipfire/ovpn/enable
>>>>>>>>> /var/ipfire/ovpn/server.conf
>>>>>>>>> /var/ipfire/ovpn/settings
>>>>>>>>> /var/ipfire/pakfire/settings
>>>>>>>>> /var/ipfire/ppp
>>>>>>>>> /var/ipfire/ppp/fake-resolv.conf
>>>>>>>>> /var/ipfire/ppp/settings
>>>>>>>>> /var/ipfire/private/cakey.pem
>>>>>>>>> /var/ipfire/proxy
>>>>>>>>> /var/ipfire/proxy/asnbl-helper.conf
>>>>>>>>> /var/ipfire/proxy/cachemgr.conf
>>>>>>>>> /var/ipfire/proxy/enable
>>>>>>>>> /var/ipfire/proxy/settings
>>>>>>>>> /var/ipfire/proxy/squid.conf
>>>>>>>>> /var/ipfire/qos/bin
>>>>>>>>> /var/ipfire/qos/bin/qos.sh
>>>>>>>>> /var/ipfire/qos/classes
>>>>>>>>> /var/ipfire/qos/level7config
>>>>>>>>> /var/ipfire/qos/portconfig
>>>>>>>>> /var/ipfire/qos/settings
>>>>>>>>> /var/ipfire/qos/subclasses
>>>>>>>>> /var/ipfire/qos/tosconfig
>>>>>>>>> /var/ipfire/remote/enablessh
>>>>>>>>> /var/ipfire/remote/settings
>>>>>>>>> /var/ipfire/sensors/settings
>>>>>>>>> /var/ipfire/suricata/oinkmaster.conf
>>>>>>>>> /var/ipfire/suricata/oinkmaster-modify-sids.conf
>>>>>>>>> /var/ipfire/suricata/oinkmaster-provider-includes.conf
>>>>>>>>> /var/ipfire/suricata/providers-settings
>>>>>>>>> /var/ipfire/suricata/settings
>>>>>>>>> /var/ipfire/suricata/suricata-default-rules.yaml
>>>>>>>>> /var/ipfire/suricata/suricata-dns-servers.yaml
>>>>>>>>> /var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
>>>>>>>>> /var/ipfire/suricata/suricata-homenet.yaml
>>>>>>>>> /var/ipfire/suricata/suricata-http-ports.yaml
>>>>>>>>> /var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml
>>>>>>>>> /var/ipfire/suricata/suricata-used-providers.yaml
>>>>>>>>> /var/ipfire/time/
>>>>>>>>> /var/ipfire/time/counter.conf
>>>>>>>>> /var/ipfire/time/enable
>>>>>>>>> /var/ipfire/time/settime.conf
>>>>>>>>> /var/ipfire/time/settings
>>>>>>>>> /var/ipfire/upnp/settings
>>>>>>>>> /var/ipfire/urlfilter
>>>>>>>>> /var/ipfire/urlfilter/settings
>>>>>>>>> /var/ipfire/urlfilter/squidGuard.conf
>>>>>>>>> /var/ipfire/vpn
>>>>>>>>> /var/ipfire/vpn/config
>>>>>>>>> /var/ipfire/vpn/ipsec.conf
>>>>>>>>> /var/ipfire/vpn/settings
>>>>>>>>> /var/ipfire/wakeonlan/clients.conf
>>>>>>>>> /var/ipfire/wio/wio.conf
>>>>>>>>> /var/ipfire/wireless/config
>>>>>>>>> /var/ipfire/wireless/settings
>>>>>>>>> /var/lib/suricata
>>>>>>>>> /var/log/rrd/collectd
>>>>>>>>> /var/log/rrd/hddshutdown-md127.rrd
>>>>>>>>> /var/log/rrd/hddshutdown-sda.rrd
>>>>>>>>> /var/log/rrd/hddshutdown-sdb.rrd
>>>>>>>>> /var/log/rrd/hddtemp-md127.rrd
>>>>>>>>> /var/log/rrd/hddtemp-sda.rrd
>>>>>>>>> /var/log/rrd/hddtemp-sdb.rrd
>>>>>>>>> /var/log/rrd/wio
>>>>>>>>> /var/log/vnstat
>>>>>>>>> /var/tmp/idsrules-emerging.tar.gz
>>>>>>>>> /var/tmp/idsrules-sslbl_blacklist.rules
>>>>>>>>> 
>>>>>>>>> Regards,
>>>>>>>>> Adolf.
>>>>>>>>>> 
>>>>>>>>>> Bernhard
>>>>>>>>>>> -Michael
>>>>>>>>>>> 
>>>>>>>>>>>> On 29 Mar 2022, at 14:10, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>> 
>>>>>>>>>>>> Just tried this patch out on my vm testbed system and it still doesn't work for me. The backup file had got to 1.3GB when I deleted the backup file as it was still growing. The normal correct backup file on that vm machine is around 7MB
>>>>>>>>>>>> 
>>>>>>>>>>>> The above was the case for both running it from the WUI or from the command line from my unprivileged user using sudo backupctrl exclude
>>>>>>>>>>>> 
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> 
>>>>>>>>>>>> Adolf.
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> On 29/03/2022 14:27, Michael Tremer wrote:
>>>>>>>>>>>>> This patch fixes globbing expansion in the backup include file list
>>>>>>>>>>>>> which got broken in c7e0d73e7cfd7be95db9d0a5f3392b8241813d5b.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
>>>>>>>>>>>>> ---
>>>>>>>>>>>>>    config/backup/backup.pl | 8 ++++----
>>>>>>>>>>>>>    1 file changed, 4 insertions(+), 4 deletions(-)
>>>>>>>>>>>>> 
>>>>>>>>>>>>> diff --git a/config/backup/backup.pl b/config/backup/backup.pl
>>>>>>>>>>>>> index a2337cf23..6f9295e94 100644
>>>>>>>>>>>>> --- a/config/backup/backup.pl
>>>>>>>>>>>>> +++ b/config/backup/backup.pl
>>>>>>>>>>>>> @@ -19,6 +19,8 @@
>>>>>>>>>>>>> # #
>>>>>>>>>>>>> ###############################################################################
>>>>>>>>>>>>>    +shopt -s nullglob
>>>>>>>>>>>>> +
>>>>>>>>>>>>>    NOW="$(date "+%Y-%m-%d-%H:%M")"
>>>>>>>>>>>>>      list_addons() {
>>>>>>>>>>>>> @@ -38,10 +40,8 @@ process_includes() {
>>>>>>>>>>>>>        for include in $@; do
>>>>>>>>>>>>>            local file
>>>>>>>>>>>>>            while read -r file; do
>>>>>>>>>>>>> -            for file in ${file}; do
>>>>>>>>>>>>> -                if [ -e "/${file}" ]; then
>>>>>>>>>>>>> -                    echo "${file}"
>>>>>>>>>>>>> -                fi
>>>>>>>>>>>>> +            for file in /${file}; do
>>>>>>>>>>>>> +                echo "${file}"
>>>>>>>>>>>>>                done
>>>>>>>>>>>>>            done < "${include}"
>>>>>>>>>>>>>        done | sort -u
>>>>>>>>>>> 
>>>>>> 
>

Adolf Belka March 30, 2022, 9:20 p.m. UTC | #18

Hi,

On 30/03/2022 17:05, Michael Tremer wrote:
> Hello Bernhard,
> 
> You are obviously right. My brain is absolute mush these days and doesn’t allow me to function.
> 
> I will correct this by the end of the day.
> 
> -Michael
> 
>> On 30 Mar 2022, at 15:38, Bernhard Bitsch <bbitsch@ipfire.org> wrote:
>>
>> Hello all,
>>
>>
>> Am 30.03.2022 um 16:06 schrieb Michael Tremer:
>>> Hello everyone,
>>> Thank you very much for helping me test this.
>>> I decided that we will have to create an emergency core update for this, because it will otherwise break people’s system. We also have a vulnerability in zlib which is being handed around by the press, so that has helped me to make a decision.
>>
>> I think this is a good solution. Shall we add a warning to backup wiki page? Because in the moment a backup -> reinstall -> restore process leaves an incomplete system.
>>
>>> I also pushed two more patches. One filters out any empty lines because they have quite a damaging effect and it happens easily that they are added to the include list. Secondly, I brought back the existence check which I thought was only there for the globbing mechanism, but also had the small side-effect of filtering out the file list early on which helps keeping tar happy.
>>> Please review those changes and please install the new c166 from testing as soon as the build has finished (I am not sure if you are all on the nightly-builds list to see any notifications).
I am not on the notifications list but I will check first thing tomorrow 
and when the latest has been updated I will do a test install and 
evaluation of the backup and provide confirmation of how it functions.

Regards,
Adolf.
>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=4f0e7f24f293ca10096e616010cfd4417afc157c
>> The test on empty string should be in the 'while read -r file' loop. The empty lines are in the include files. The outer for loop iterates over the file list.
>>
>> Regards,
>> Bernhard
>>
>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=b275771fdd10ac7116e5d400dc10d3148df8ac27
>>> Best,
>>> -Michael
>>>> On 29 Mar 2022, at 20:21, Bernhard Bitsch <bbitsch@ipfire.org> wrote:
>>>>
>>>> Just checked the blank line problem.
>>>> Pattern is empty , file name list is '/' --> no names produced ( with existence check )
>>>> Pattern is empty --> produced file name '/' ( without -e check )
>>>>
>>>> Am 29.03.2022 um 20:36 schrieb Bernhard Bitsch:
>>>>> Hi all,
>>>>> Am 29.03.2022 um 17:23 schrieb Adolf Belka:
>>>>>> Hi All,
>>>>>>
>>>>>> On 29/03/2022 16:22, Michael Tremer wrote:
>>>>>>>
>>>>>>>> On 29 Mar 2022, at 15:22, Bernhard Bitsch <bbitsch@ipfire.org> wrote:
>>>>>>>>
>>>>>>>> Looks like your include.user file contains absolute paths.
>>>>>> That was not the problem but I have corrected that. I would have expected that when the main include and exclude files were made relative that the update script would also have checked for any entries in the include.user and exclude.user files but it looks like it didn't.th
>>>>> According to the source they are chosen.
>>>>>>>
>>>>>>> Or an empty line maybe?
>>>>>> That turned out to be the problem. There was an empty line after the two lines I had added. You only saw it if you scrolled the pointer down the file in the editor.
>>>>> Interesting. A blank line should produce nothing.
>>>>>>
>>>>>> Removing that blank line removed the line only with a / and the backup then successfully ran and created a backup file containing all the directories expected.
>>>>>>
>>>>>> The only thing I found is that there was the following tar error message
>>>>>>
>>>>>> tar: Exiting with failure status due to previous errors
>>>>>>
>>>>>> The only thing I found in the backup output was several lines such as
>>>>>>
>>>>>> tar: /etc/squid/squid.conf.local: Cannot stat: No such file or directory
>>>>>> tar: /etc/squid/squid.conf.pre.local: Cannot stat: No such file or directory
>>>>>>
>>>>>> There were seven lines in total like this and the three I checked were files that did not exist on my system. I presume that these are files that should be backed up if present but don't have to be present so that the error messages are nothing to worry about.
>>>>>> There were no error messages when doing the backup via the WUI.
>>>>>>
>>>>> Good find. ;)
>>>>> Michael's patch removes the test of existence, globbing produces existent file names only. But I suppose there are several full names of files, not existing in each implementation.
>>>>> Conclusion: Repair the file globbing by the shopt and do the existence check. So all existent files are chosen, that are listed in the include files.
>>>>> Regards,
>>>>> Bernhard
>>>>>> The blank line was only on my vm testbed system but as I clone this for any testing activities I will have to check all of my existing vm's to fix this.
>>>>>>
>>>>>> Glad we resolved this simply. Sorry for the anxiety.
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Adolf.
>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Am 29.03.2022 um 16:14 schrieb Adolf Belka:
>>>>>>>>> Hi All,
>>>>>>>>> This is what I get with the pushd/popd version in place on the same system
>>>>>>>>> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
>>>>>>>>> etc/group
>>>>>>>>> etc/hosts
>>>>>>>>> etc/hosts.allow
>>>>>>>>> etc/hosts.deny
>>>>>>>>> etc/httpd/server.crt
>>>>>>>>> etc/httpd/server.csr
>>>>>>>>> etc/httpd/server-ecdsa.crt
>>>>>>>>> etc/httpd/server-ecdsa.csr
>>>>>>>>> etc/httpd/server-ecdsa.key
>>>>>>>>> etc/httpd/server.key
>>>>>>>>> etc/ipsec.user.conf
>>>>>>>>> etc/ipsec.user.secrets
>>>>>>>>> etc/logrotate.d
>>>>>>>>> etc/passwd
>>>>>>>>> etc/shadow
>>>>>>>>> etc/ssh/sshd_config
>>>>>>>>> etc/ssh/ssh_host_ecdsa_key
>>>>>>>>> etc/ssh/ssh_host_ecdsa_key.pub
>>>>>>>>> etc/ssh/ssh_host_ed25519_key
>>>>>>>>> etc/ssh/ssh_host_ed25519_key.pub
>>>>>>>>> etc/ssh/ssh_host_rsa_key
>>>>>>>>> etc/ssh/ssh_host_rsa_key.pub
>>>>>>>>> /etc/sudoers
>>>>>>>>> etc/sysconfig/createfiles
>>>>>>>>> etc/sysconfig/firewall.local
>>>>>>>>> etc/sysconfig/lm_sensors
>>>>>>>>> etc/sysconfig/modules
>>>>>>>>> etc/sysconfig/ramdisk
>>>>>>>>> etc/sysconfig/rc
>>>>>>>>> etc/sysconfig/rc.local
>>>>>>>>> etc/unbound
>>>>>>>>> /home/ahb
>>>>>>>>> root/.bash_history
>>>>>>>>> var/ipfire/accounting/settings.conf
>>>>>>>>> var/ipfire/auth/users
>>>>>>>>> var/ipfire/backup/addons/backup
>>>>>>>>> var/ipfire/backup/exclude.user
>>>>>>>>> var/ipfire/backup/include.user
>>>>>>>>> var/ipfire/ca/cacert.pem
>>>>>>>>> var/ipfire/captive/agb.txt
>>>>>>>>> var/ipfire/captive/clients
>>>>>>>>> var/ipfire/captive/coupons
>>>>>>>>> var/ipfire/captive/logo.dat
>>>>>>>>> var/ipfire/captive/settings
>>>>>>>>> var/ipfire/captive/terms.txt
>>>>>>>>> var/ipfire/captive/voucher_out
>>>>>>>>> var/ipfire/certs/hostcert.pem
>>>>>>>>> var/ipfire/certs/hostkey.pem
>>>>>>>>> var/ipfire/certs/phoebevmipseccert.pem
>>>>>>>>> var/ipfire/connscheduler/connscheduler.conf
>>>>>>>>> var/ipfire/crls/cacrl.pem
>>>>>>>>> var/ipfire/cups/cups-browsed.conf
>>>>>>>>> var/ipfire/cups/subscriptions.conf
>>>>>>>>> var/ipfire/ddns/config
>>>>>>>>> var/ipfire/ddns/ddns.conf
>>>>>>>>> var/ipfire/ddns/settings
>>>>>>>>> var/ipfire/dhcp/advoptions
>>>>>>>>> var/ipfire/dhcp/advoptions-list
>>>>>>>>> var/ipfire/dhcpc/dhcpcd.conf
>>>>>>>>> var/ipfire/dhcp/dhcpd.conf
>>>>>>>>> var/ipfire/dhcp/dhcpd.conf.local
>>>>>>>>> var/ipfire/dhcp/enable_blue
>>>>>>>>> var/ipfire/dhcp/enable_green
>>>>>>>>> var/ipfire/dhcp/fixleases
>>>>>>>>> var/ipfire/dhcp/settings
>>>>>>>>> var/ipfire/dma/auth.conf
>>>>>>>>> var/ipfire/dma/dma.conf
>>>>>>>>> var/ipfire/dma/mail.conf
>>>>>>>>> var/ipfire/dns
>>>>>>>>> var/ipfire/dnsforward/config
>>>>>>>>> var/ipfire/dns/settings
>>>>>>>>> var/ipfire/ethernet/aliases
>>>>>>>>> var/ipfire/ethernet/settings
>>>>>>>>> var/ipfire/ethernet/wireless
>>>>>>>>> var/ipfire/extrahd/settings
>>>>>>>>> var/ipfire/firewall
>>>>>>>>> var/ipfire/firewall/config
>>>>>>>>> var/ipfire/firewall/settings
>>>>>>>>> var/ipfire/fwhosts
>>>>>>>>> var/ipfire/isdn/settings
>>>>>>>>> var/ipfire/logging/settings
>>>>>>>>> var/ipfire/mac/settings
>>>>>>>>> var/ipfire/main/firstsetup_ok
>>>>>>>>> var/ipfire/main/gpl_accepted
>>>>>>>>> var/ipfire/main/hostname.conf
>>>>>>>>> var/ipfire/main/hosts
>>>>>>>>> var/ipfire/main/manualpages
>>>>>>>>> var/ipfire/main/routing
>>>>>>>>> var/ipfire/main/security
>>>>>>>>> var/ipfire/main/send_profile
>>>>>>>>> var/ipfire/main/settings
>>>>>>>>> var/ipfire/modem/settings
>>>>>>>>> var/ipfire/optionsfw/settings
>>>>>>>>> var/ipfire/ovpn
>>>>>>>>> var/ipfire/ovpn/ccd.conf
>>>>>>>>> var/ipfire/ovpn/collectd.vpn
>>>>>>>>> var/ipfire/ovpn/enable
>>>>>>>>> var/ipfire/ovpn/server.conf
>>>>>>>>> var/ipfire/ovpn/settings
>>>>>>>>> var/ipfire/pakfire/settings
>>>>>>>>> var/ipfire/ppp
>>>>>>>>> var/ipfire/ppp/fake-resolv.conf
>>>>>>>>> var/ipfire/ppp/settings
>>>>>>>>> var/ipfire/private/cakey.pem
>>>>>>>>> var/ipfire/proxy
>>>>>>>>> var/ipfire/proxy/asnbl-helper.conf
>>>>>>>>> var/ipfire/proxy/cachemgr.conf
>>>>>>>>> var/ipfire/proxy/enable
>>>>>>>>> var/ipfire/proxy/settings
>>>>>>>>> var/ipfire/proxy/squid.conf
>>>>>>>>> var/ipfire/qos/bin
>>>>>>>>> var/ipfire/qos/classes
>>>>>>>>> var/ipfire/qos/level7config
>>>>>>>>> var/ipfire/qos/portconfig
>>>>>>>>> var/ipfire/qos/settings
>>>>>>>>> var/ipfire/qos/subclasses
>>>>>>>>> var/ipfire/qos/tosconfig
>>>>>>>>> var/ipfire/remote/enablessh
>>>>>>>>> var/ipfire/remote/settings
>>>>>>>>> var/ipfire/sensors/settings
>>>>>>>>> var/ipfire/suricata/oinkmaster.conf
>>>>>>>>> var/ipfire/suricata/oinkmaster-modify-sids.conf
>>>>>>>>> var/ipfire/suricata/oinkmaster-provider-includes.conf
>>>>>>>>> var/ipfire/suricata/providers-settings
>>>>>>>>> var/ipfire/suricata/settings
>>>>>>>>> var/ipfire/suricata/suricata-default-rules.yaml
>>>>>>>>> var/ipfire/suricata/suricata-dns-servers.yaml
>>>>>>>>> var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
>>>>>>>>> var/ipfire/suricata/suricata-homenet.yaml
>>>>>>>>> var/ipfire/suricata/suricata-http-ports.yaml
>>>>>>>>> var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml
>>>>>>>>> var/ipfire/suricata/suricata-used-providers.yaml
>>>>>>>>> var/ipfire/time/
>>>>>>>>> var/ipfire/time/counter.conf
>>>>>>>>> var/ipfire/time/enable
>>>>>>>>> var/ipfire/time/settime.conf
>>>>>>>>> var/ipfire/time/settings
>>>>>>>>> var/ipfire/upnp/settings
>>>>>>>>> var/ipfire/urlfilter
>>>>>>>>> var/ipfire/urlfilter/settings
>>>>>>>>> var/ipfire/urlfilter/squidGuard.conf
>>>>>>>>> var/ipfire/vpn
>>>>>>>>> var/ipfire/vpn/config
>>>>>>>>> var/ipfire/vpn/ipsec.conf
>>>>>>>>> var/ipfire/vpn/settings
>>>>>>>>> var/ipfire/wakeonlan/clients.conf
>>>>>>>>> var/ipfire/wio/wio.conf
>>>>>>>>> var/ipfire/wireless/config
>>>>>>>>> var/ipfire/wireless/settings
>>>>>>>>> var/lib/suricata
>>>>>>>>> var/log/rrd/collectd
>>>>>>>>> var/log/rrd/hddshutdown-md127.rrd
>>>>>>>>> var/log/rrd/hddshutdown-sda.rrd
>>>>>>>>> var/log/rrd/hddshutdown-sdb.rrd
>>>>>>>>> var/log/rrd/hddtemp-md127.rrd
>>>>>>>>> var/log/rrd/hddtemp-sda.rrd
>>>>>>>>> var/log/rrd/hddtemp-sdb.rrd
>>>>>>>>> var/log/rrd/wio
>>>>>>>>> var/log/vnstat
>>>>>>>>> var/tmp/idsrules-emerging.tar.gz
>>>>>>>>> var/tmp/idsrules-sslbl_blacklist.rules
>>>>>>>>> The following are in the previous list but not in this one:-
>>>>>>>>> /
>>>>>>>>> /etc/conntrackd/conntrackd.conf
>>>>>>>>> /etc/ipsec.user-post.conf
>>>>>>>>> /root/.gitconfig
>>>>>>>>> /root/.ssh
>>>>>>>>> Regards,
>>>>>>>>> Adolf.
>>>>>>>>> On 29/03/2022 16:01, Adolf Belka wrote:
>>>>>>>>>> Hi All,
>>>>>>>>>>
>>>>>>>>>> On 29/03/2022 15:36, Bernhard Bitsch wrote:
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> Am 29.03.2022 um 15:11 schrieb Michael Tremer:
>>>>>>>>>>>> Hello,
>>>>>>>>>>>>
>>>>>>>>>>>> Could you please send the file listing to find out what is being included what shouldn’t?
>>>>>>>>>> Unfortunately, as I stopped the backup continuing once it had reached 1.2GB, the file created was not able to be opened. Probably stopping the backup corrupted it in some way.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Just for easy quick handling, you can use
>>>>>>>>>>> '/var/ipfire/backup/bin/backup.pl list' to show the files included.
>>>>>>>>>> I gave this a go and here is the output from the command.
>>>>>>>>>>
>>>>>>>>>> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
>>>>>>>>>> /
>>>>>>>>>> /etc/conntrackd/conntrackd.conf
>>>>>>>>>> /etc/group
>>>>>>>>>> /etc/hosts
>>>>>>>>>> /etc/hosts.allow
>>>>>>>>>> /etc/hosts.deny
>>>>>>>>>> /etc/httpd/server.crt
>>>>>>>>>> /etc/httpd/server.csr
>>>>>>>>>> /etc/httpd/server-ecdsa.crt
>>>>>>>>>> /etc/httpd/server-ecdsa.csr
>>>>>>>>>> /etc/httpd/server-ecdsa.key
>>>>>>>>>> /etc/httpd/server.key
>>>>>>>>>> /etc/ipsec.user.conf
>>>>>>>>>> /etc/ipsec.user-post.conf
>>>>>>>>>> /etc/ipsec.user.secrets
>>>>>>>>>> /etc/logrotate.d
>>>>>>>>>> /etc/passwd
>>>>>>>>>> /etc/shadow
>>>>>>>>>> /etc/squid/squid.conf.local
>>>>>>>>>> /etc/squid/squid.conf.pre.local
>>>>>>>>>> /etc/ssh/sshd_config
>>>>>>>>>> /etc/ssh/ssh_host_ecdsa_key
>>>>>>>>>> /etc/ssh/ssh_host_ecdsa_key.pub
>>>>>>>>>> /etc/ssh/ssh_host_ed25519_key
>>>>>>>>>> /etc/ssh/ssh_host_ed25519_key.pub
>>>>>>>>>> /etc/ssh/ssh_host_rsa_key
>>>>>>>>>> /etc/ssh/ssh_host_rsa_key.pub
>>>>>>>>>> //etc/sudoers
>>>>>>>>>> /etc/sysconfig/createfiles
>>>>>>>>>> /etc/sysconfig/firewall.local
>>>>>>>>>> /etc/sysconfig/lm_sensors
>>>>>>>>>> /etc/sysconfig/modules
>>>>>>>>>> /etc/sysconfig/ramdisk
>>>>>>>>>> /etc/sysconfig/rc
>>>>>>>>>> /etc/sysconfig/rc.local
>>>>>>>>>> /etc/unbound
>>>>>>>>>> //home/ahb
>>>>>>>>>> /root/.bash_history
>>>>>>>>>> /root/.gitconfig
>>>>>>>>>> /root/.ssh
>>>>>>>>>> /var/ipfire/accounting/settings.conf
>>>>>>>>>> /var/ipfire/auth/users
>>>>>>>>>> /var/ipfire/backup/addons/backup
>>>>>>>>>> /var/ipfire/backup/exclude.user
>>>>>>>>>> /var/ipfire/backup/include.user
>>>>>>>>>> /var/ipfire/ca/cacert.pem
>>>>>>>>>> /var/ipfire/captive/agb.txt
>>>>>>>>>> /var/ipfire/captive/clients
>>>>>>>>>> /var/ipfire/captive/coupons
>>>>>>>>>> /var/ipfire/captive/logo.dat
>>>>>>>>>> /var/ipfire/captive/settings
>>>>>>>>>> /var/ipfire/captive/terms.txt
>>>>>>>>>> /var/ipfire/captive/voucher_out
>>>>>>>>>> /var/ipfire/certs/hostcert.pem
>>>>>>>>>> /var/ipfire/certs/hostkey.pem
>>>>>>>>>> /var/ipfire/certs/phoebevmipseccert.pem
>>>>>>>>>> /var/ipfire/connscheduler/connscheduler.conf
>>>>>>>>>> /var/ipfire/crls/cacrl.pem
>>>>>>>>>> /var/ipfire/cups/cups-browsed.conf
>>>>>>>>>> /var/ipfire/cups/subscriptions.conf
>>>>>>>>>> /var/ipfire/ddns/config
>>>>>>>>>> /var/ipfire/ddns/ddns.conf
>>>>>>>>>> /var/ipfire/ddns/settings
>>>>>>>>>> /var/ipfire/dhcp/advoptions
>>>>>>>>>> /var/ipfire/dhcp/advoptions-list
>>>>>>>>>> /var/ipfire/dhcpc/dhcpcd.conf
>>>>>>>>>> /var/ipfire/dhcp/dhcpd.conf
>>>>>>>>>> /var/ipfire/dhcp/dhcpd.conf.local
>>>>>>>>>> /var/ipfire/dhcp/enable_blue
>>>>>>>>>> /var/ipfire/dhcp/enable_green
>>>>>>>>>> /var/ipfire/dhcp/fixleases
>>>>>>>>>> /var/ipfire/dhcp/settings
>>>>>>>>>> /var/ipfire/dma/auth.conf
>>>>>>>>>> /var/ipfire/dma/dma.conf
>>>>>>>>>> /var/ipfire/dma/mail.conf
>>>>>>>>>> /var/ipfire/dns
>>>>>>>>>> /var/ipfire/dnsforward/config
>>>>>>>>>> /var/ipfire/dns/settings
>>>>>>>>>> /var/ipfire/ethernet/aliases
>>>>>>>>>> /var/ipfire/ethernet/settings
>>>>>>>>>> /var/ipfire/ethernet/wireless
>>>>>>>>>> /var/ipfire/extrahd/settings
>>>>>>>>>> /var/ipfire/firewall
>>>>>>>>>> /var/ipfire/firewall/config
>>>>>>>>>> /var/ipfire/firewall/settings
>>>>>>>>>> /var/ipfire/fwhosts
>>>>>>>>>> /var/ipfire/isdn/settings
>>>>>>>>>> /var/ipfire/logging/settings
>>>>>>>>>> /var/ipfire/mac/settings
>>>>>>>>>> /var/ipfire/main/firstsetup_ok
>>>>>>>>>> /var/ipfire/main/gpl_accepted
>>>>>>>>>> /var/ipfire/main/hostname.conf
>>>>>>>>>> /var/ipfire/main/hosts
>>>>>>>>>> /var/ipfire/main/manualpages
>>>>>>>>>> /var/ipfire/main/routing
>>>>>>>>>> /var/ipfire/main/security
>>>>>>>>>> /var/ipfire/main/send_profile
>>>>>>>>>> /var/ipfire/main/settings
>>>>>>>>>> /var/ipfire/modem/settings
>>>>>>>>>> /var/ipfire/optionsfw/settings
>>>>>>>>>> /var/ipfire/ovpn
>>>>>>>>>> /var/ipfire/ovpn/ccd.conf
>>>>>>>>>> /var/ipfire/ovpn/collectd.vpn
>>>>>>>>>> /var/ipfire/ovpn/enable
>>>>>>>>>> /var/ipfire/ovpn/server.conf
>>>>>>>>>> /var/ipfire/ovpn/settings
>>>>>>>>>> /var/ipfire/pakfire/settings
>>>>>>>>>> /var/ipfire/ppp
>>>>>>>>>> /var/ipfire/ppp/fake-resolv.conf
>>>>>>>>>> /var/ipfire/ppp/settings
>>>>>>>>>> /var/ipfire/private/cakey.pem
>>>>>>>>>> /var/ipfire/proxy
>>>>>>>>>> /var/ipfire/proxy/asnbl-helper.conf
>>>>>>>>>> /var/ipfire/proxy/cachemgr.conf
>>>>>>>>>> /var/ipfire/proxy/enable
>>>>>>>>>> /var/ipfire/proxy/settings
>>>>>>>>>> /var/ipfire/proxy/squid.conf
>>>>>>>>>> /var/ipfire/qos/bin
>>>>>>>>>> /var/ipfire/qos/bin/qos.sh
>>>>>>>>>> /var/ipfire/qos/classes
>>>>>>>>>> /var/ipfire/qos/level7config
>>>>>>>>>> /var/ipfire/qos/portconfig
>>>>>>>>>> /var/ipfire/qos/settings
>>>>>>>>>> /var/ipfire/qos/subclasses
>>>>>>>>>> /var/ipfire/qos/tosconfig
>>>>>>>>>> /var/ipfire/remote/enablessh
>>>>>>>>>> /var/ipfire/remote/settings
>>>>>>>>>> /var/ipfire/sensors/settings
>>>>>>>>>> /var/ipfire/suricata/oinkmaster.conf
>>>>>>>>>> /var/ipfire/suricata/oinkmaster-modify-sids.conf
>>>>>>>>>> /var/ipfire/suricata/oinkmaster-provider-includes.conf
>>>>>>>>>> /var/ipfire/suricata/providers-settings
>>>>>>>>>> /var/ipfire/suricata/settings
>>>>>>>>>> /var/ipfire/suricata/suricata-default-rules.yaml
>>>>>>>>>> /var/ipfire/suricata/suricata-dns-servers.yaml
>>>>>>>>>> /var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
>>>>>>>>>> /var/ipfire/suricata/suricata-homenet.yaml
>>>>>>>>>> /var/ipfire/suricata/suricata-http-ports.yaml
>>>>>>>>>> /var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml
>>>>>>>>>> /var/ipfire/suricata/suricata-used-providers.yaml
>>>>>>>>>> /var/ipfire/time/
>>>>>>>>>> /var/ipfire/time/counter.conf
>>>>>>>>>> /var/ipfire/time/enable
>>>>>>>>>> /var/ipfire/time/settime.conf
>>>>>>>>>> /var/ipfire/time/settings
>>>>>>>>>> /var/ipfire/upnp/settings
>>>>>>>>>> /var/ipfire/urlfilter
>>>>>>>>>> /var/ipfire/urlfilter/settings
>>>>>>>>>> /var/ipfire/urlfilter/squidGuard.conf
>>>>>>>>>> /var/ipfire/vpn
>>>>>>>>>> /var/ipfire/vpn/config
>>>>>>>>>> /var/ipfire/vpn/ipsec.conf
>>>>>>>>>> /var/ipfire/vpn/settings
>>>>>>>>>> /var/ipfire/wakeonlan/clients.conf
>>>>>>>>>> /var/ipfire/wio/wio.conf
>>>>>>>>>> /var/ipfire/wireless/config
>>>>>>>>>> /var/ipfire/wireless/settings
>>>>>>>>>> /var/lib/suricata
>>>>>>>>>> /var/log/rrd/collectd
>>>>>>>>>> /var/log/rrd/hddshutdown-md127.rrd
>>>>>>>>>> /var/log/rrd/hddshutdown-sda.rrd
>>>>>>>>>> /var/log/rrd/hddshutdown-sdb.rrd
>>>>>>>>>> /var/log/rrd/hddtemp-md127.rrd
>>>>>>>>>> /var/log/rrd/hddtemp-sda.rrd
>>>>>>>>>> /var/log/rrd/hddtemp-sdb.rrd
>>>>>>>>>> /var/log/rrd/wio
>>>>>>>>>> /var/log/vnstat
>>>>>>>>>> /var/tmp/idsrules-emerging.tar.gz
>>>>>>>>>> /var/tmp/idsrules-sslbl_blacklist.rules
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Adolf.
>>>>>>>>>>>
>>>>>>>>>>> Bernhard
>>>>>>>>>>>> -Michael
>>>>>>>>>>>>
>>>>>>>>>>>>> On 29 Mar 2022, at 14:10, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Just tried this patch out on my vm testbed system and it still doesn't work for me. The backup file had got to 1.3GB when I deleted the backup file as it was still growing. The normal correct backup file on that vm machine is around 7MB
>>>>>>>>>>>>>
>>>>>>>>>>>>> The above was the case for both running it from the WUI or from the command line from my unprivileged user using sudo backupctrl exclude
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 29/03/2022 14:27, Michael Tremer wrote:
>>>>>>>>>>>>>> This patch fixes globbing expansion in the backup include file list
>>>>>>>>>>>>>> which got broken in c7e0d73e7cfd7be95db9d0a5f3392b8241813d5b.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>     config/backup/backup.pl | 8 ++++----
>>>>>>>>>>>>>>     1 file changed, 4 insertions(+), 4 deletions(-)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> diff --git a/config/backup/backup.pl b/config/backup/backup.pl
>>>>>>>>>>>>>> index a2337cf23..6f9295e94 100644
>>>>>>>>>>>>>> --- a/config/backup/backup.pl
>>>>>>>>>>>>>> +++ b/config/backup/backup.pl
>>>>>>>>>>>>>> @@ -19,6 +19,8 @@
>>>>>>>>>>>>>> # #
>>>>>>>>>>>>>> ###############################################################################
>>>>>>>>>>>>>>     +shopt -s nullglob
>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>     NOW="$(date "+%Y-%m-%d-%H:%M")"
>>>>>>>>>>>>>>       list_addons() {
>>>>>>>>>>>>>> @@ -38,10 +40,8 @@ process_includes() {
>>>>>>>>>>>>>>         for include in $@; do
>>>>>>>>>>>>>>             local file
>>>>>>>>>>>>>>             while read -r file; do
>>>>>>>>>>>>>> -            for file in ${file}; do
>>>>>>>>>>>>>> -                if [ -e "/${file}" ]; then
>>>>>>>>>>>>>> -                    echo "${file}"
>>>>>>>>>>>>>> -                fi
>>>>>>>>>>>>>> +            for file in /${file}; do
>>>>>>>>>>>>>> +                echo "${file}"
>>>>>>>>>>>>>>                 done
>>>>>>>>>>>>>>             done < "${include}"
>>>>>>>>>>>>>>         done | sort -u
>>>>>>>>>>>>
>>>>>>>
>

Adolf Belka March 31, 2022, 7:41 a.m. UTC | #19

Hi Michael,

On 30/03/2022 23:20, Adolf Belka wrote:
> Hi,
>
> On 30/03/2022 17:05, Michael Tremer wrote:
>> Hello Bernhard,
>>
>> You are obviously right. My brain is absolute mush these days and doesn’t allow me to function.
>>
>> I will correct this by the end of the day.
>>
>> -Michael
>>
>>> On 30 Mar 2022, at 15:38, Bernhard Bitsch <bbitsch@ipfire.org> wrote:
>>>
>>> Hello all,
>>>
>>>
>>> Am 30.03.2022 um 16:06 schrieb Michael Tremer:
>>>> Hello everyone,
>>>> Thank you very much for helping me test this.
>>>> I decided that we will have to create an emergency core update for this, because it will otherwise break people’s system. We also have a vulnerability in zlib which is being handed around by the press, so that has helped me to make a decision.
>>>
>>> I think this is a good solution. Shall we add a warning to backup wiki page? Because in the moment a backup -> reinstall -> restore process leaves an incomplete system.
>>>
>>>> I also pushed two more patches. One filters out any empty lines because they have quite a damaging effect and it happens easily that they are added to the include list. Secondly, I brought back the existence check which I thought was only there for the globbing mechanism, but also had the small side-effect of filtering out the file list early on which helps keeping tar happy.
>>>> Please review those changes and please install the new c166 from testing as soon as the build has finished (I am not sure if you are all on the nightly-builds list to see any notifications).
> I am not on the notifications list but I will check first thing tomorrow and when the latest has been updated I will do a test install and evaluation of the backup and provide confirmation of how it functions.
>
As CU166 has been moved to master I took a clone of my CU165 vm and then did a CU on Testing with it rather than doing a fresh install. If you would also want a test of a fresh install done I can also do that.

The version I tested was IPFire 2.27 (x86_64) - Core Update 166 Development Build: master/8f696f60

Anyway the backup is working fine again. All the directories are present. The size of the backup is now very similar to previously also confirming all contents should be good.

I also added back in a couple of blank lines into my include.user file and the backup worked without any problems.

I also did a restore of the backup and again as far as I can tell with a quick check through on my testbed, everything is fine.

So everything looks good to me.

Regards,

Adolf.

> Regards,
> Adolf.
>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=4f0e7f24f293ca10096e616010cfd4417afc157c
>>> The test on empty string should be in the 'while read -r file' loop. The empty lines are in the include files. The outer for loop iterates over the file list.
>>>
>>> Regards,
>>> Bernhard
>>>
>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=b275771fdd10ac7116e5d400dc10d3148df8ac27
>>>> Best,
>>>> -Michael
>>>>> On 29 Mar 2022, at 20:21, Bernhard Bitsch <bbitsch@ipfire.org> wrote:
>>>>>
>>>>> Just checked the blank line problem.
>>>>> Pattern is empty , file name list is '/' --> no names produced ( with existence check )
>>>>> Pattern is empty --> produced file name '/' ( without -e check )
>>>>>
>>>>> Am 29.03.2022 um 20:36 schrieb Bernhard Bitsch:
>>>>>> Hi all,
>>>>>> Am 29.03.2022 um 17:23 schrieb Adolf Belka:
>>>>>>> Hi All,
>>>>>>>
>>>>>>> On 29/03/2022 16:22, Michael Tremer wrote:
>>>>>>>>
>>>>>>>>> On 29 Mar 2022, at 15:22, Bernhard Bitsch <bbitsch@ipfire.org> wrote:
>>>>>>>>>
>>>>>>>>> Looks like your include.user file contains absolute paths.
>>>>>>> That was not the problem but I have corrected that. I would have expected that when the main include and exclude files were made relative that the update script would also have checked for any entries in the include.user and exclude.user files but it looks like it didn't.th
>>>>>> According to the source they are chosen.
>>>>>>>>
>>>>>>>> Or an empty line maybe?
>>>>>>> That turned out to be the problem. There was an empty line after the two lines I had added. You only saw it if you scrolled the pointer down the file in the editor.
>>>>>> Interesting. A blank line should produce nothing.
>>>>>>>
>>>>>>> Removing that blank line removed the line only with a / and the backup then successfully ran and created a backup file containing all the directories expected.
>>>>>>>
>>>>>>> The only thing I found is that there was the following tar error message
>>>>>>>
>>>>>>> tar: Exiting with failure status due to previous errors
>>>>>>>
>>>>>>> The only thing I found in the backup output was several lines such as
>>>>>>>
>>>>>>> tar: /etc/squid/squid.conf.local: Cannot stat: No such file or directory
>>>>>>> tar: /etc/squid/squid.conf.pre.local: Cannot stat: No such file or directory
>>>>>>>
>>>>>>> There were seven lines in total like this and the three I checked were files that did not exist on my system. I presume that these are files that should be backed up if present but don't have to be present so that the error messages are nothing to worry about.
>>>>>>> There were no error messages when doing the backup via the WUI.
>>>>>>>
>>>>>> Good find. ;)
>>>>>> Michael's patch removes the test of existence, globbing produces existent file names only. But I suppose there are several full names of files, not existing in each implementation.
>>>>>> Conclusion: Repair the file globbing by the shopt and do the existence check. So all existent files are chosen, that are listed in the include files.
>>>>>> Regards,
>>>>>> Bernhard
>>>>>>> The blank line was only on my vm testbed system but as I clone this for any testing activities I will have to check all of my existing vm's to fix this.
>>>>>>>
>>>>>>> Glad we resolved this simply. Sorry for the anxiety.
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Adolf.
>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Am 29.03.2022 um 16:14 schrieb Adolf Belka:
>>>>>>>>>> Hi All,
>>>>>>>>>> This is what I get with the pushd/popd version in place on the same system
>>>>>>>>>> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
>>>>>>>>>> etc/group
>>>>>>>>>> etc/hosts
>>>>>>>>>> etc/hosts.allow
>>>>>>>>>> etc/hosts.deny
>>>>>>>>>> etc/httpd/server.crt
>>>>>>>>>> etc/httpd/server.csr
>>>>>>>>>> etc/httpd/server-ecdsa.crt
>>>>>>>>>> etc/httpd/server-ecdsa.csr
>>>>>>>>>> etc/httpd/server-ecdsa.key
>>>>>>>>>> etc/httpd/server.key
>>>>>>>>>> etc/ipsec.user.conf
>>>>>>>>>> etc/ipsec.user.secrets
>>>>>>>>>> etc/logrotate.d
>>>>>>>>>> etc/passwd
>>>>>>>>>> etc/shadow
>>>>>>>>>> etc/ssh/sshd_config
>>>>>>>>>> etc/ssh/ssh_host_ecdsa_key
>>>>>>>>>> etc/ssh/ssh_host_ecdsa_key.pub
>>>>>>>>>> etc/ssh/ssh_host_ed25519_key
>>>>>>>>>> etc/ssh/ssh_host_ed25519_key.pub
>>>>>>>>>> etc/ssh/ssh_host_rsa_key
>>>>>>>>>> etc/ssh/ssh_host_rsa_key.pub
>>>>>>>>>> /etc/sudoers
>>>>>>>>>> etc/sysconfig/createfiles
>>>>>>>>>> etc/sysconfig/firewall.local
>>>>>>>>>> etc/sysconfig/lm_sensors
>>>>>>>>>> etc/sysconfig/modules
>>>>>>>>>> etc/sysconfig/ramdisk
>>>>>>>>>> etc/sysconfig/rc
>>>>>>>>>> etc/sysconfig/rc.local
>>>>>>>>>> etc/unbound
>>>>>>>>>> /home/ahb
>>>>>>>>>> root/.bash_history
>>>>>>>>>> var/ipfire/accounting/settings.conf
>>>>>>>>>> var/ipfire/auth/users
>>>>>>>>>> var/ipfire/backup/addons/backup
>>>>>>>>>> var/ipfire/backup/exclude.user
>>>>>>>>>> var/ipfire/backup/include.user
>>>>>>>>>> var/ipfire/ca/cacert.pem
>>>>>>>>>> var/ipfire/captive/agb.txt
>>>>>>>>>> var/ipfire/captive/clients
>>>>>>>>>> var/ipfire/captive/coupons
>>>>>>>>>> var/ipfire/captive/logo.dat
>>>>>>>>>> var/ipfire/captive/settings
>>>>>>>>>> var/ipfire/captive/terms.txt
>>>>>>>>>> var/ipfire/captive/voucher_out
>>>>>>>>>> var/ipfire/certs/hostcert.pem
>>>>>>>>>> var/ipfire/certs/hostkey.pem
>>>>>>>>>> var/ipfire/certs/phoebevmipseccert.pem
>>>>>>>>>> var/ipfire/connscheduler/connscheduler.conf
>>>>>>>>>> var/ipfire/crls/cacrl.pem
>>>>>>>>>> var/ipfire/cups/cups-browsed.conf
>>>>>>>>>> var/ipfire/cups/subscriptions.conf
>>>>>>>>>> var/ipfire/ddns/config
>>>>>>>>>> var/ipfire/ddns/ddns.conf
>>>>>>>>>> var/ipfire/ddns/settings
>>>>>>>>>> var/ipfire/dhcp/advoptions
>>>>>>>>>> var/ipfire/dhcp/advoptions-list
>>>>>>>>>> var/ipfire/dhcpc/dhcpcd.conf
>>>>>>>>>> var/ipfire/dhcp/dhcpd.conf
>>>>>>>>>> var/ipfire/dhcp/dhcpd.conf.local
>>>>>>>>>> var/ipfire/dhcp/enable_blue
>>>>>>>>>> var/ipfire/dhcp/enable_green
>>>>>>>>>> var/ipfire/dhcp/fixleases
>>>>>>>>>> var/ipfire/dhcp/settings
>>>>>>>>>> var/ipfire/dma/auth.conf
>>>>>>>>>> var/ipfire/dma/dma.conf
>>>>>>>>>> var/ipfire/dma/mail.conf
>>>>>>>>>> var/ipfire/dns
>>>>>>>>>> var/ipfire/dnsforward/config
>>>>>>>>>> var/ipfire/dns/settings
>>>>>>>>>> var/ipfire/ethernet/aliases
>>>>>>>>>> var/ipfire/ethernet/settings
>>>>>>>>>> var/ipfire/ethernet/wireless
>>>>>>>>>> var/ipfire/extrahd/settings
>>>>>>>>>> var/ipfire/firewall
>>>>>>>>>> var/ipfire/firewall/config
>>>>>>>>>> var/ipfire/firewall/settings
>>>>>>>>>> var/ipfire/fwhosts
>>>>>>>>>> var/ipfire/isdn/settings
>>>>>>>>>> var/ipfire/logging/settings
>>>>>>>>>> var/ipfire/mac/settings
>>>>>>>>>> var/ipfire/main/firstsetup_ok
>>>>>>>>>> var/ipfire/main/gpl_accepted
>>>>>>>>>> var/ipfire/main/hostname.conf
>>>>>>>>>> var/ipfire/main/hosts
>>>>>>>>>> var/ipfire/main/manualpages
>>>>>>>>>> var/ipfire/main/routing
>>>>>>>>>> var/ipfire/main/security
>>>>>>>>>> var/ipfire/main/send_profile
>>>>>>>>>> var/ipfire/main/settings
>>>>>>>>>> var/ipfire/modem/settings
>>>>>>>>>> var/ipfire/optionsfw/settings
>>>>>>>>>> var/ipfire/ovpn
>>>>>>>>>> var/ipfire/ovpn/ccd.conf
>>>>>>>>>> var/ipfire/ovpn/collectd.vpn
>>>>>>>>>> var/ipfire/ovpn/enable
>>>>>>>>>> var/ipfire/ovpn/server.conf
>>>>>>>>>> var/ipfire/ovpn/settings
>>>>>>>>>> var/ipfire/pakfire/settings
>>>>>>>>>> var/ipfire/ppp
>>>>>>>>>> var/ipfire/ppp/fake-resolv.conf
>>>>>>>>>> var/ipfire/ppp/settings
>>>>>>>>>> var/ipfire/private/cakey.pem
>>>>>>>>>> var/ipfire/proxy
>>>>>>>>>> var/ipfire/proxy/asnbl-helper.conf
>>>>>>>>>> var/ipfire/proxy/cachemgr.conf
>>>>>>>>>> var/ipfire/proxy/enable
>>>>>>>>>> var/ipfire/proxy/settings
>>>>>>>>>> var/ipfire/proxy/squid.conf
>>>>>>>>>> var/ipfire/qos/bin
>>>>>>>>>> var/ipfire/qos/classes
>>>>>>>>>> var/ipfire/qos/level7config
>>>>>>>>>> var/ipfire/qos/portconfig
>>>>>>>>>> var/ipfire/qos/settings
>>>>>>>>>> var/ipfire/qos/subclasses
>>>>>>>>>> var/ipfire/qos/tosconfig
>>>>>>>>>> var/ipfire/remote/enablessh
>>>>>>>>>> var/ipfire/remote/settings
>>>>>>>>>> var/ipfire/sensors/settings
>>>>>>>>>> var/ipfire/suricata/oinkmaster.conf
>>>>>>>>>> var/ipfire/suricata/oinkmaster-modify-sids.conf
>>>>>>>>>> var/ipfire/suricata/oinkmaster-provider-includes.conf
>>>>>>>>>> var/ipfire/suricata/providers-settings
>>>>>>>>>> var/ipfire/suricata/settings
>>>>>>>>>> var/ipfire/suricata/suricata-default-rules.yaml
>>>>>>>>>> var/ipfire/suricata/suricata-dns-servers.yaml
>>>>>>>>>> var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
>>>>>>>>>> var/ipfire/suricata/suricata-homenet.yaml
>>>>>>>>>> var/ipfire/suricata/suricata-http-ports.yaml
>>>>>>>>>> var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml
>>>>>>>>>> var/ipfire/suricata/suricata-used-providers.yaml
>>>>>>>>>> var/ipfire/time/
>>>>>>>>>> var/ipfire/time/counter.conf
>>>>>>>>>> var/ipfire/time/enable
>>>>>>>>>> var/ipfire/time/settime.conf
>>>>>>>>>> var/ipfire/time/settings
>>>>>>>>>> var/ipfire/upnp/settings
>>>>>>>>>> var/ipfire/urlfilter
>>>>>>>>>> var/ipfire/urlfilter/settings
>>>>>>>>>> var/ipfire/urlfilter/squidGuard.conf
>>>>>>>>>> var/ipfire/vpn
>>>>>>>>>> var/ipfire/vpn/config
>>>>>>>>>> var/ipfire/vpn/ipsec.conf
>>>>>>>>>> var/ipfire/vpn/settings
>>>>>>>>>> var/ipfire/wakeonlan/clients.conf
>>>>>>>>>> var/ipfire/wio/wio.conf
>>>>>>>>>> var/ipfire/wireless/config
>>>>>>>>>> var/ipfire/wireless/settings
>>>>>>>>>> var/lib/suricata
>>>>>>>>>> var/log/rrd/collectd
>>>>>>>>>> var/log/rrd/hddshutdown-md127.rrd
>>>>>>>>>> var/log/rrd/hddshutdown-sda.rrd
>>>>>>>>>> var/log/rrd/hddshutdown-sdb.rrd
>>>>>>>>>> var/log/rrd/hddtemp-md127.rrd
>>>>>>>>>> var/log/rrd/hddtemp-sda.rrd
>>>>>>>>>> var/log/rrd/hddtemp-sdb.rrd
>>>>>>>>>> var/log/rrd/wio
>>>>>>>>>> var/log/vnstat
>>>>>>>>>> var/tmp/idsrules-emerging.tar.gz
>>>>>>>>>> var/tmp/idsrules-sslbl_blacklist.rules
>>>>>>>>>> The following are in the previous list but not in this one:-
>>>>>>>>>> /
>>>>>>>>>> /etc/conntrackd/conntrackd.conf
>>>>>>>>>> /etc/ipsec.user-post.conf
>>>>>>>>>> /root/.gitconfig
>>>>>>>>>> /root/.ssh
>>>>>>>>>> Regards,
>>>>>>>>>> Adolf.
>>>>>>>>>> On 29/03/2022 16:01, Adolf Belka wrote:
>>>>>>>>>>> Hi All,
>>>>>>>>>>>
>>>>>>>>>>> On 29/03/2022 15:36, Bernhard Bitsch wrote:
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> Am 29.03.2022 um 15:11 schrieb Michael Tremer:
>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Could you please send the file listing to find out what is being included what shouldn’t?
>>>>>>>>>>> Unfortunately, as I stopped the backup continuing once it had reached 1.2GB, the file created was not able to be opened. Probably stopping the backup corrupted it in some way.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Just for easy quick handling, you can use
>>>>>>>>>>>> '/var/ipfire/backup/bin/backup.pl list' to show the files included.
>>>>>>>>>>> I gave this a go and here is the output from the command.
>>>>>>>>>>>
>>>>>>>>>>> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
>>>>>>>>>>> /
>>>>>>>>>>> /etc/conntrackd/conntrackd.conf
>>>>>>>>>>> /etc/group
>>>>>>>>>>> /etc/hosts
>>>>>>>>>>> /etc/hosts.allow
>>>>>>>>>>> /etc/hosts.deny
>>>>>>>>>>> /etc/httpd/server.crt
>>>>>>>>>>> /etc/httpd/server.csr
>>>>>>>>>>> /etc/httpd/server-ecdsa.crt
>>>>>>>>>>> /etc/httpd/server-ecdsa.csr
>>>>>>>>>>> /etc/httpd/server-ecdsa.key
>>>>>>>>>>> /etc/httpd/server.key
>>>>>>>>>>> /etc/ipsec.user.conf
>>>>>>>>>>> /etc/ipsec.user-post.conf
>>>>>>>>>>> /etc/ipsec.user.secrets
>>>>>>>>>>> /etc/logrotate.d
>>>>>>>>>>> /etc/passwd
>>>>>>>>>>> /etc/shadow
>>>>>>>>>>> /etc/squid/squid.conf.local
>>>>>>>>>>> /etc/squid/squid.conf.pre.local
>>>>>>>>>>> /etc/ssh/sshd_config
>>>>>>>>>>> /etc/ssh/ssh_host_ecdsa_key
>>>>>>>>>>> /etc/ssh/ssh_host_ecdsa_key.pub
>>>>>>>>>>> /etc/ssh/ssh_host_ed25519_key
>>>>>>>>>>> /etc/ssh/ssh_host_ed25519_key.pub
>>>>>>>>>>> /etc/ssh/ssh_host_rsa_key
>>>>>>>>>>> /etc/ssh/ssh_host_rsa_key.pub
>>>>>>>>>>> //etc/sudoers
>>>>>>>>>>> /etc/sysconfig/createfiles
>>>>>>>>>>> /etc/sysconfig/firewall.local
>>>>>>>>>>> /etc/sysconfig/lm_sensors
>>>>>>>>>>> /etc/sysconfig/modules
>>>>>>>>>>> /etc/sysconfig/ramdisk
>>>>>>>>>>> /etc/sysconfig/rc
>>>>>>>>>>> /etc/sysconfig/rc.local
>>>>>>>>>>> /etc/unbound
>>>>>>>>>>> //home/ahb
>>>>>>>>>>> /root/.bash_history
>>>>>>>>>>> /root/.gitconfig
>>>>>>>>>>> /root/.ssh
>>>>>>>>>>> /var/ipfire/accounting/settings.conf
>>>>>>>>>>> /var/ipfire/auth/users
>>>>>>>>>>> /var/ipfire/backup/addons/backup
>>>>>>>>>>> /var/ipfire/backup/exclude.user
>>>>>>>>>>> /var/ipfire/backup/include.user
>>>>>>>>>>> /var/ipfire/ca/cacert.pem
>>>>>>>>>>> /var/ipfire/captive/agb.txt
>>>>>>>>>>> /var/ipfire/captive/clients
>>>>>>>>>>> /var/ipfire/captive/coupons
>>>>>>>>>>> /var/ipfire/captive/logo.dat
>>>>>>>>>>> /var/ipfire/captive/settings
>>>>>>>>>>> /var/ipfire/captive/terms.txt
>>>>>>>>>>> /var/ipfire/captive/voucher_out
>>>>>>>>>>> /var/ipfire/certs/hostcert.pem
>>>>>>>>>>> /var/ipfire/certs/hostkey.pem
>>>>>>>>>>> /var/ipfire/certs/phoebevmipseccert.pem
>>>>>>>>>>> /var/ipfire/connscheduler/connscheduler.conf
>>>>>>>>>>> /var/ipfire/crls/cacrl.pem
>>>>>>>>>>> /var/ipfire/cups/cups-browsed.conf
>>>>>>>>>>> /var/ipfire/cups/subscriptions.conf
>>>>>>>>>>> /var/ipfire/ddns/config
>>>>>>>>>>> /var/ipfire/ddns/ddns.conf
>>>>>>>>>>> /var/ipfire/ddns/settings
>>>>>>>>>>> /var/ipfire/dhcp/advoptions
>>>>>>>>>>> /var/ipfire/dhcp/advoptions-list
>>>>>>>>>>> /var/ipfire/dhcpc/dhcpcd.conf
>>>>>>>>>>> /var/ipfire/dhcp/dhcpd.conf
>>>>>>>>>>> /var/ipfire/dhcp/dhcpd.conf.local
>>>>>>>>>>> /var/ipfire/dhcp/enable_blue
>>>>>>>>>>> /var/ipfire/dhcp/enable_green
>>>>>>>>>>> /var/ipfire/dhcp/fixleases
>>>>>>>>>>> /var/ipfire/dhcp/settings
>>>>>>>>>>> /var/ipfire/dma/auth.conf
>>>>>>>>>>> /var/ipfire/dma/dma.conf
>>>>>>>>>>> /var/ipfire/dma/mail.conf
>>>>>>>>>>> /var/ipfire/dns
>>>>>>>>>>> /var/ipfire/dnsforward/config
>>>>>>>>>>> /var/ipfire/dns/settings
>>>>>>>>>>> /var/ipfire/ethernet/aliases
>>>>>>>>>>> /var/ipfire/ethernet/settings
>>>>>>>>>>> /var/ipfire/ethernet/wireless
>>>>>>>>>>> /var/ipfire/extrahd/settings
>>>>>>>>>>> /var/ipfire/firewall
>>>>>>>>>>> /var/ipfire/firewall/config
>>>>>>>>>>> /var/ipfire/firewall/settings
>>>>>>>>>>> /var/ipfire/fwhosts
>>>>>>>>>>> /var/ipfire/isdn/settings
>>>>>>>>>>> /var/ipfire/logging/settings
>>>>>>>>>>> /var/ipfire/mac/settings
>>>>>>>>>>> /var/ipfire/main/firstsetup_ok
>>>>>>>>>>> /var/ipfire/main/gpl_accepted
>>>>>>>>>>> /var/ipfire/main/hostname.conf
>>>>>>>>>>> /var/ipfire/main/hosts
>>>>>>>>>>> /var/ipfire/main/manualpages
>>>>>>>>>>> /var/ipfire/main/routing
>>>>>>>>>>> /var/ipfire/main/security
>>>>>>>>>>> /var/ipfire/main/send_profile
>>>>>>>>>>> /var/ipfire/main/settings
>>>>>>>>>>> /var/ipfire/modem/settings
>>>>>>>>>>> /var/ipfire/optionsfw/settings
>>>>>>>>>>> /var/ipfire/ovpn
>>>>>>>>>>> /var/ipfire/ovpn/ccd.conf
>>>>>>>>>>> /var/ipfire/ovpn/collectd.vpn
>>>>>>>>>>> /var/ipfire/ovpn/enable
>>>>>>>>>>> /var/ipfire/ovpn/server.conf
>>>>>>>>>>> /var/ipfire/ovpn/settings
>>>>>>>>>>> /var/ipfire/pakfire/settings
>>>>>>>>>>> /var/ipfire/ppp
>>>>>>>>>>> /var/ipfire/ppp/fake-resolv.conf
>>>>>>>>>>> /var/ipfire/ppp/settings
>>>>>>>>>>> /var/ipfire/private/cakey.pem
>>>>>>>>>>> /var/ipfire/proxy
>>>>>>>>>>> /var/ipfire/proxy/asnbl-helper.conf
>>>>>>>>>>> /var/ipfire/proxy/cachemgr.conf
>>>>>>>>>>> /var/ipfire/proxy/enable
>>>>>>>>>>> /var/ipfire/proxy/settings
>>>>>>>>>>> /var/ipfire/proxy/squid.conf
>>>>>>>>>>> /var/ipfire/qos/bin
>>>>>>>>>>> /var/ipfire/qos/bin/qos.sh
>>>>>>>>>>> /var/ipfire/qos/classes
>>>>>>>>>>> /var/ipfire/qos/level7config
>>>>>>>>>>> /var/ipfire/qos/portconfig
>>>>>>>>>>> /var/ipfire/qos/settings
>>>>>>>>>>> /var/ipfire/qos/subclasses
>>>>>>>>>>> /var/ipfire/qos/tosconfig
>>>>>>>>>>> /var/ipfire/remote/enablessh
>>>>>>>>>>> /var/ipfire/remote/settings
>>>>>>>>>>> /var/ipfire/sensors/settings
>>>>>>>>>>> /var/ipfire/suricata/oinkmaster.conf
>>>>>>>>>>> /var/ipfire/suricata/oinkmaster-modify-sids.conf
>>>>>>>>>>> /var/ipfire/suricata/oinkmaster-provider-includes.conf
>>>>>>>>>>> /var/ipfire/suricata/providers-settings
>>>>>>>>>>> /var/ipfire/suricata/settings
>>>>>>>>>>> /var/ipfire/suricata/suricata-default-rules.yaml
>>>>>>>>>>> /var/ipfire/suricata/suricata-dns-servers.yaml
>>>>>>>>>>> /var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
>>>>>>>>>>> /var/ipfire/suricata/suricata-homenet.yaml
>>>>>>>>>>> /var/ipfire/suricata/suricata-http-ports.yaml
>>>>>>>>>>> /var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml
>>>>>>>>>>> /var/ipfire/suricata/suricata-used-providers.yaml
>>>>>>>>>>> /var/ipfire/time/
>>>>>>>>>>> /var/ipfire/time/counter.conf
>>>>>>>>>>> /var/ipfire/time/enable
>>>>>>>>>>> /var/ipfire/time/settime.conf
>>>>>>>>>>> /var/ipfire/time/settings
>>>>>>>>>>> /var/ipfire/upnp/settings
>>>>>>>>>>> /var/ipfire/urlfilter
>>>>>>>>>>> /var/ipfire/urlfilter/settings
>>>>>>>>>>> /var/ipfire/urlfilter/squidGuard.conf
>>>>>>>>>>> /var/ipfire/vpn
>>>>>>>>>>> /var/ipfire/vpn/config
>>>>>>>>>>> /var/ipfire/vpn/ipsec.conf
>>>>>>>>>>> /var/ipfire/vpn/settings
>>>>>>>>>>> /var/ipfire/wakeonlan/clients.conf
>>>>>>>>>>> /var/ipfire/wio/wio.conf
>>>>>>>>>>> /var/ipfire/wireless/config
>>>>>>>>>>> /var/ipfire/wireless/settings
>>>>>>>>>>> /var/lib/suricata
>>>>>>>>>>> /var/log/rrd/collectd
>>>>>>>>>>> /var/log/rrd/hddshutdown-md127.rrd
>>>>>>>>>>> /var/log/rrd/hddshutdown-sda.rrd
>>>>>>>>>>> /var/log/rrd/hddshutdown-sdb.rrd
>>>>>>>>>>> /var/log/rrd/hddtemp-md127.rrd
>>>>>>>>>>> /var/log/rrd/hddtemp-sda.rrd
>>>>>>>>>>> /var/log/rrd/hddtemp-sdb.rrd
>>>>>>>>>>> /var/log/rrd/wio
>>>>>>>>>>> /var/log/vnstat
>>>>>>>>>>> /var/tmp/idsrules-emerging.tar.gz
>>>>>>>>>>> /var/tmp/idsrules-sslbl_blacklist.rules
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>> Adolf.
>>>>>>>>>>>>
>>>>>>>>>>>> Bernhard
>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 29 Mar 2022, at 14:10, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Just tried this patch out on my vm testbed system and it still doesn't work for me. The backup file had got to 1.3GB when I deleted the backup file as it was still growing. The normal correct backup file on that vm machine is around 7MB
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The above was the case for both running it from the WUI or from the command line from my unprivileged user using sudo backupctrl exclude
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 29/03/2022 14:27, Michael Tremer wrote:
>>>>>>>>>>>>>>> This patch fixes globbing expansion in the backup include file list
>>>>>>>>>>>>>>> which got broken in c7e0d73e7cfd7be95db9d0a5f3392b8241813d5b.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>     config/backup/backup.pl | 8 ++++----
>>>>>>>>>>>>>>>     1 file changed, 4 insertions(+), 4 deletions(-)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> diff --git a/config/backup/backup.pl b/config/backup/backup.pl
>>>>>>>>>>>>>>> index a2337cf23..6f9295e94 100644
>>>>>>>>>>>>>>> --- a/config/backup/backup.pl
>>>>>>>>>>>>>>> +++ b/config/backup/backup.pl
>>>>>>>>>>>>>>> @@ -19,6 +19,8 @@
>>>>>>>>>>>>>>> # #
>>>>>>>>>>>>>>> ###############################################################################
>>>>>>>>>>>>>>>     +shopt -s nullglob
>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>     NOW="$(date "+%Y-%m-%d-%H:%M")"
>>>>>>>>>>>>>>>       list_addons() {
>>>>>>>>>>>>>>> @@ -38,10 +40,8 @@ process_includes() {
>>>>>>>>>>>>>>>         for include in $@; do
>>>>>>>>>>>>>>>             local file
>>>>>>>>>>>>>>>             while read -r file; do
>>>>>>>>>>>>>>> -            for file in ${file}; do
>>>>>>>>>>>>>>> -                if [ -e "/${file}" ]; then
>>>>>>>>>>>>>>> -                    echo "${file}"
>>>>>>>>>>>>>>> -                fi
>>>>>>>>>>>>>>> +            for file in /${file}; do
>>>>>>>>>>>>>>> +                echo "${file}"
>>>>>>>>>>>>>>>                 done
>>>>>>>>>>>>>>>             done < "${include}"
>>>>>>>>>>>>>>>         done | sort -u
>>>>>>>>>>>>>
>>>>>>>>
>>
>

Michael Tremer March 31, 2022, 7:46 a.m. UTC | #20

Hello Adolf,

Brilliant. Thank you very much for testing this in detail.

Best,
-Michael

> On 31 Mar 2022, at 08:41, Adolf Belka <adolf.belka@ipfire.org> wrote:
> 
> Hi Michael,
> 
> On 30/03/2022 23:20, Adolf Belka wrote:
>> Hi,
>> 
>> On 30/03/2022 17:05, Michael Tremer wrote:
>>> Hello Bernhard,
>>> 
>>> You are obviously right. My brain is absolute mush these days and doesn’t allow me to function.
>>> 
>>> I will correct this by the end of the day.
>>> 
>>> -Michael
>>> 
>>>> On 30 Mar 2022, at 15:38, Bernhard Bitsch <bbitsch@ipfire.org> wrote:
>>>> 
>>>> Hello all,
>>>> 
>>>> 
>>>> Am 30.03.2022 um 16:06 schrieb Michael Tremer:
>>>>> Hello everyone,
>>>>> Thank you very much for helping me test this.
>>>>> I decided that we will have to create an emergency core update for this, because it will otherwise break people’s system. We also have a vulnerability in zlib which is being handed around by the press, so that has helped me to make a decision.
>>>> 
>>>> I think this is a good solution. Shall we add a warning to backup wiki page? Because in the moment a backup -> reinstall -> restore process leaves an incomplete system.
>>>> 
>>>>> I also pushed two more patches. One filters out any empty lines because they have quite a damaging effect and it happens easily that they are added to the include list. Secondly, I brought back the existence check which I thought was only there for the globbing mechanism, but also had the small side-effect of filtering out the file list early on which helps keeping tar happy.
>>>>> Please review those changes and please install the new c166 from testing as soon as the build has finished (I am not sure if you are all on the nightly-builds list to see any notifications).
>> I am not on the notifications list but I will check first thing tomorrow and when the latest has been updated I will do a test install and evaluation of the backup and provide confirmation of how it functions.
>> 
> As CU166 has been moved to master I took a clone of my CU165 vm and then did a CU on Testing with it rather than doing a fresh install. If you would also want a test of a fresh install done I can also do that.
> 
> The version I tested was IPFire 2.27 (x86_64) - Core Update 166 Development Build: master/8f696f60
> 
> Anyway the backup is working fine again. All the directories are present. The size of the backup is now very similar to previously also confirming all contents should be good.
> 
> I also added back in a couple of blank lines into my include.user file and the backup worked without any problems.
> 
> I also did a restore of the backup and again as far as I can tell with a quick check through on my testbed, everything is fine.
> 
> So everything looks good to me.
> 
> Regards,
> 
> Adolf.
> 
>> Regards,
>> Adolf.
>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=4f0e7f24f293ca10096e616010cfd4417afc157c
>>>> The test on empty string should be in the 'while read -r file' loop. The empty lines are in the include files. The outer for loop iterates over the file list.
>>>> 
>>>> Regards,
>>>> Bernhard
>>>> 
>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=b275771fdd10ac7116e5d400dc10d3148df8ac27
>>>>> Best,
>>>>> -Michael
>>>>>> On 29 Mar 2022, at 20:21, Bernhard Bitsch <bbitsch@ipfire.org> wrote:
>>>>>> 
>>>>>> Just checked the blank line problem.
>>>>>> Pattern is empty , file name list is '/' --> no names produced ( with existence check )
>>>>>> Pattern is empty --> produced file name '/' ( without -e check )
>>>>>> 
>>>>>> Am 29.03.2022 um 20:36 schrieb Bernhard Bitsch:
>>>>>>> Hi all,
>>>>>>> Am 29.03.2022 um 17:23 schrieb Adolf Belka:
>>>>>>>> Hi All,
>>>>>>>> 
>>>>>>>> On 29/03/2022 16:22, Michael Tremer wrote:
>>>>>>>>> 
>>>>>>>>>> On 29 Mar 2022, at 15:22, Bernhard Bitsch <bbitsch@ipfire.org> wrote:
>>>>>>>>>> 
>>>>>>>>>> Looks like your include.user file contains absolute paths.
>>>>>>>> That was not the problem but I have corrected that. I would have expected that when the main include and exclude files were made relative that the update script would also have checked for any entries in the include.user and exclude.user files but it looks like it didn't.th
>>>>>>> According to the source they are chosen.
>>>>>>>>> 
>>>>>>>>> Or an empty line maybe?
>>>>>>>> That turned out to be the problem. There was an empty line after the two lines I had added. You only saw it if you scrolled the pointer down the file in the editor.
>>>>>>> Interesting. A blank line should produce nothing.
>>>>>>>> 
>>>>>>>> Removing that blank line removed the line only with a / and the backup then successfully ran and created a backup file containing all the directories expected.
>>>>>>>> 
>>>>>>>> The only thing I found is that there was the following tar error message
>>>>>>>> 
>>>>>>>> tar: Exiting with failure status due to previous errors
>>>>>>>> 
>>>>>>>> The only thing I found in the backup output was several lines such as
>>>>>>>> 
>>>>>>>> tar: /etc/squid/squid.conf.local: Cannot stat: No such file or directory
>>>>>>>> tar: /etc/squid/squid.conf.pre.local: Cannot stat: No such file or directory
>>>>>>>> 
>>>>>>>> There were seven lines in total like this and the three I checked were files that did not exist on my system. I presume that these are files that should be backed up if present but don't have to be present so that the error messages are nothing to worry about.
>>>>>>>> There were no error messages when doing the backup via the WUI.
>>>>>>>> 
>>>>>>> Good find. ;)
>>>>>>> Michael's patch removes the test of existence, globbing produces existent file names only. But I suppose there are several full names of files, not existing in each implementation.
>>>>>>> Conclusion: Repair the file globbing by the shopt and do the existence check. So all existent files are chosen, that are listed in the include files.
>>>>>>> Regards,
>>>>>>> Bernhard
>>>>>>>> The blank line was only on my vm testbed system but as I clone this for any testing activities I will have to check all of my existing vm's to fix this.
>>>>>>>> 
>>>>>>>> Glad we resolved this simply. Sorry for the anxiety.
>>>>>>>> 
>>>>>>>> Regards,
>>>>>>>> 
>>>>>>>> Adolf.
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Am 29.03.2022 um 16:14 schrieb Adolf Belka:
>>>>>>>>>>> Hi All,
>>>>>>>>>>> This is what I get with the pushd/popd version in place on the same system
>>>>>>>>>>> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
>>>>>>>>>>> etc/group
>>>>>>>>>>> etc/hosts
>>>>>>>>>>> etc/hosts.allow
>>>>>>>>>>> etc/hosts.deny
>>>>>>>>>>> etc/httpd/server.crt
>>>>>>>>>>> etc/httpd/server.csr
>>>>>>>>>>> etc/httpd/server-ecdsa.crt
>>>>>>>>>>> etc/httpd/server-ecdsa.csr
>>>>>>>>>>> etc/httpd/server-ecdsa.key
>>>>>>>>>>> etc/httpd/server.key
>>>>>>>>>>> etc/ipsec.user.conf
>>>>>>>>>>> etc/ipsec.user.secrets
>>>>>>>>>>> etc/logrotate.d
>>>>>>>>>>> etc/passwd
>>>>>>>>>>> etc/shadow
>>>>>>>>>>> etc/ssh/sshd_config
>>>>>>>>>>> etc/ssh/ssh_host_ecdsa_key
>>>>>>>>>>> etc/ssh/ssh_host_ecdsa_key.pub
>>>>>>>>>>> etc/ssh/ssh_host_ed25519_key
>>>>>>>>>>> etc/ssh/ssh_host_ed25519_key.pub
>>>>>>>>>>> etc/ssh/ssh_host_rsa_key
>>>>>>>>>>> etc/ssh/ssh_host_rsa_key.pub
>>>>>>>>>>> /etc/sudoers
>>>>>>>>>>> etc/sysconfig/createfiles
>>>>>>>>>>> etc/sysconfig/firewall.local
>>>>>>>>>>> etc/sysconfig/lm_sensors
>>>>>>>>>>> etc/sysconfig/modules
>>>>>>>>>>> etc/sysconfig/ramdisk
>>>>>>>>>>> etc/sysconfig/rc
>>>>>>>>>>> etc/sysconfig/rc.local
>>>>>>>>>>> etc/unbound
>>>>>>>>>>> /home/ahb
>>>>>>>>>>> root/.bash_history
>>>>>>>>>>> var/ipfire/accounting/settings.conf
>>>>>>>>>>> var/ipfire/auth/users
>>>>>>>>>>> var/ipfire/backup/addons/backup
>>>>>>>>>>> var/ipfire/backup/exclude.user
>>>>>>>>>>> var/ipfire/backup/include.user
>>>>>>>>>>> var/ipfire/ca/cacert.pem
>>>>>>>>>>> var/ipfire/captive/agb.txt
>>>>>>>>>>> var/ipfire/captive/clients
>>>>>>>>>>> var/ipfire/captive/coupons
>>>>>>>>>>> var/ipfire/captive/logo.dat
>>>>>>>>>>> var/ipfire/captive/settings
>>>>>>>>>>> var/ipfire/captive/terms.txt
>>>>>>>>>>> var/ipfire/captive/voucher_out
>>>>>>>>>>> var/ipfire/certs/hostcert.pem
>>>>>>>>>>> var/ipfire/certs/hostkey.pem
>>>>>>>>>>> var/ipfire/certs/phoebevmipseccert.pem
>>>>>>>>>>> var/ipfire/connscheduler/connscheduler.conf
>>>>>>>>>>> var/ipfire/crls/cacrl.pem
>>>>>>>>>>> var/ipfire/cups/cups-browsed.conf
>>>>>>>>>>> var/ipfire/cups/subscriptions.conf
>>>>>>>>>>> var/ipfire/ddns/config
>>>>>>>>>>> var/ipfire/ddns/ddns.conf
>>>>>>>>>>> var/ipfire/ddns/settings
>>>>>>>>>>> var/ipfire/dhcp/advoptions
>>>>>>>>>>> var/ipfire/dhcp/advoptions-list
>>>>>>>>>>> var/ipfire/dhcpc/dhcpcd.conf
>>>>>>>>>>> var/ipfire/dhcp/dhcpd.conf
>>>>>>>>>>> var/ipfire/dhcp/dhcpd.conf.local
>>>>>>>>>>> var/ipfire/dhcp/enable_blue
>>>>>>>>>>> var/ipfire/dhcp/enable_green
>>>>>>>>>>> var/ipfire/dhcp/fixleases
>>>>>>>>>>> var/ipfire/dhcp/settings
>>>>>>>>>>> var/ipfire/dma/auth.conf
>>>>>>>>>>> var/ipfire/dma/dma.conf
>>>>>>>>>>> var/ipfire/dma/mail.conf
>>>>>>>>>>> var/ipfire/dns
>>>>>>>>>>> var/ipfire/dnsforward/config
>>>>>>>>>>> var/ipfire/dns/settings
>>>>>>>>>>> var/ipfire/ethernet/aliases
>>>>>>>>>>> var/ipfire/ethernet/settings
>>>>>>>>>>> var/ipfire/ethernet/wireless
>>>>>>>>>>> var/ipfire/extrahd/settings
>>>>>>>>>>> var/ipfire/firewall
>>>>>>>>>>> var/ipfire/firewall/config
>>>>>>>>>>> var/ipfire/firewall/settings
>>>>>>>>>>> var/ipfire/fwhosts
>>>>>>>>>>> var/ipfire/isdn/settings
>>>>>>>>>>> var/ipfire/logging/settings
>>>>>>>>>>> var/ipfire/mac/settings
>>>>>>>>>>> var/ipfire/main/firstsetup_ok
>>>>>>>>>>> var/ipfire/main/gpl_accepted
>>>>>>>>>>> var/ipfire/main/hostname.conf
>>>>>>>>>>> var/ipfire/main/hosts
>>>>>>>>>>> var/ipfire/main/manualpages
>>>>>>>>>>> var/ipfire/main/routing
>>>>>>>>>>> var/ipfire/main/security
>>>>>>>>>>> var/ipfire/main/send_profile
>>>>>>>>>>> var/ipfire/main/settings
>>>>>>>>>>> var/ipfire/modem/settings
>>>>>>>>>>> var/ipfire/optionsfw/settings
>>>>>>>>>>> var/ipfire/ovpn
>>>>>>>>>>> var/ipfire/ovpn/ccd.conf
>>>>>>>>>>> var/ipfire/ovpn/collectd.vpn
>>>>>>>>>>> var/ipfire/ovpn/enable
>>>>>>>>>>> var/ipfire/ovpn/server.conf
>>>>>>>>>>> var/ipfire/ovpn/settings
>>>>>>>>>>> var/ipfire/pakfire/settings
>>>>>>>>>>> var/ipfire/ppp
>>>>>>>>>>> var/ipfire/ppp/fake-resolv.conf
>>>>>>>>>>> var/ipfire/ppp/settings
>>>>>>>>>>> var/ipfire/private/cakey.pem
>>>>>>>>>>> var/ipfire/proxy
>>>>>>>>>>> var/ipfire/proxy/asnbl-helper.conf
>>>>>>>>>>> var/ipfire/proxy/cachemgr.conf
>>>>>>>>>>> var/ipfire/proxy/enable
>>>>>>>>>>> var/ipfire/proxy/settings
>>>>>>>>>>> var/ipfire/proxy/squid.conf
>>>>>>>>>>> var/ipfire/qos/bin
>>>>>>>>>>> var/ipfire/qos/classes
>>>>>>>>>>> var/ipfire/qos/level7config
>>>>>>>>>>> var/ipfire/qos/portconfig
>>>>>>>>>>> var/ipfire/qos/settings
>>>>>>>>>>> var/ipfire/qos/subclasses
>>>>>>>>>>> var/ipfire/qos/tosconfig
>>>>>>>>>>> var/ipfire/remote/enablessh
>>>>>>>>>>> var/ipfire/remote/settings
>>>>>>>>>>> var/ipfire/sensors/settings
>>>>>>>>>>> var/ipfire/suricata/oinkmaster.conf
>>>>>>>>>>> var/ipfire/suricata/oinkmaster-modify-sids.conf
>>>>>>>>>>> var/ipfire/suricata/oinkmaster-provider-includes.conf
>>>>>>>>>>> var/ipfire/suricata/providers-settings
>>>>>>>>>>> var/ipfire/suricata/settings
>>>>>>>>>>> var/ipfire/suricata/suricata-default-rules.yaml
>>>>>>>>>>> var/ipfire/suricata/suricata-dns-servers.yaml
>>>>>>>>>>> var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
>>>>>>>>>>> var/ipfire/suricata/suricata-homenet.yaml
>>>>>>>>>>> var/ipfire/suricata/suricata-http-ports.yaml
>>>>>>>>>>> var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml
>>>>>>>>>>> var/ipfire/suricata/suricata-used-providers.yaml
>>>>>>>>>>> var/ipfire/time/
>>>>>>>>>>> var/ipfire/time/counter.conf
>>>>>>>>>>> var/ipfire/time/enable
>>>>>>>>>>> var/ipfire/time/settime.conf
>>>>>>>>>>> var/ipfire/time/settings
>>>>>>>>>>> var/ipfire/upnp/settings
>>>>>>>>>>> var/ipfire/urlfilter
>>>>>>>>>>> var/ipfire/urlfilter/settings
>>>>>>>>>>> var/ipfire/urlfilter/squidGuard.conf
>>>>>>>>>>> var/ipfire/vpn
>>>>>>>>>>> var/ipfire/vpn/config
>>>>>>>>>>> var/ipfire/vpn/ipsec.conf
>>>>>>>>>>> var/ipfire/vpn/settings
>>>>>>>>>>> var/ipfire/wakeonlan/clients.conf
>>>>>>>>>>> var/ipfire/wio/wio.conf
>>>>>>>>>>> var/ipfire/wireless/config
>>>>>>>>>>> var/ipfire/wireless/settings
>>>>>>>>>>> var/lib/suricata
>>>>>>>>>>> var/log/rrd/collectd
>>>>>>>>>>> var/log/rrd/hddshutdown-md127.rrd
>>>>>>>>>>> var/log/rrd/hddshutdown-sda.rrd
>>>>>>>>>>> var/log/rrd/hddshutdown-sdb.rrd
>>>>>>>>>>> var/log/rrd/hddtemp-md127.rrd
>>>>>>>>>>> var/log/rrd/hddtemp-sda.rrd
>>>>>>>>>>> var/log/rrd/hddtemp-sdb.rrd
>>>>>>>>>>> var/log/rrd/wio
>>>>>>>>>>> var/log/vnstat
>>>>>>>>>>> var/tmp/idsrules-emerging.tar.gz
>>>>>>>>>>> var/tmp/idsrules-sslbl_blacklist.rules
>>>>>>>>>>> The following are in the previous list but not in this one:-
>>>>>>>>>>> /
>>>>>>>>>>> /etc/conntrackd/conntrackd.conf
>>>>>>>>>>> /etc/ipsec.user-post.conf
>>>>>>>>>>> /root/.gitconfig
>>>>>>>>>>> /root/.ssh
>>>>>>>>>>> Regards,
>>>>>>>>>>> Adolf.
>>>>>>>>>>> On 29/03/2022 16:01, Adolf Belka wrote:
>>>>>>>>>>>> Hi All,
>>>>>>>>>>>> 
>>>>>>>>>>>> On 29/03/2022 15:36, Bernhard Bitsch wrote:
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Am 29.03.2022 um 15:11 schrieb Michael Tremer:
>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Could you please send the file listing to find out what is being included what shouldn’t?
>>>>>>>>>>>> Unfortunately, as I stopped the backup continuing once it had reached 1.2GB, the file created was not able to be opened. Probably stopping the backup corrupted it in some way.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Just for easy quick handling, you can use
>>>>>>>>>>>>> '/var/ipfire/backup/bin/backup.pl list' to show the files included.
>>>>>>>>>>>> I gave this a go and here is the output from the command.
>>>>>>>>>>>> 
>>>>>>>>>>>> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
>>>>>>>>>>>> /
>>>>>>>>>>>> /etc/conntrackd/conntrackd.conf
>>>>>>>>>>>> /etc/group
>>>>>>>>>>>> /etc/hosts
>>>>>>>>>>>> /etc/hosts.allow
>>>>>>>>>>>> /etc/hosts.deny
>>>>>>>>>>>> /etc/httpd/server.crt
>>>>>>>>>>>> /etc/httpd/server.csr
>>>>>>>>>>>> /etc/httpd/server-ecdsa.crt
>>>>>>>>>>>> /etc/httpd/server-ecdsa.csr
>>>>>>>>>>>> /etc/httpd/server-ecdsa.key
>>>>>>>>>>>> /etc/httpd/server.key
>>>>>>>>>>>> /etc/ipsec.user.conf
>>>>>>>>>>>> /etc/ipsec.user-post.conf
>>>>>>>>>>>> /etc/ipsec.user.secrets
>>>>>>>>>>>> /etc/logrotate.d
>>>>>>>>>>>> /etc/passwd
>>>>>>>>>>>> /etc/shadow
>>>>>>>>>>>> /etc/squid/squid.conf.local
>>>>>>>>>>>> /etc/squid/squid.conf.pre.local
>>>>>>>>>>>> /etc/ssh/sshd_config
>>>>>>>>>>>> /etc/ssh/ssh_host_ecdsa_key
>>>>>>>>>>>> /etc/ssh/ssh_host_ecdsa_key.pub
>>>>>>>>>>>> /etc/ssh/ssh_host_ed25519_key
>>>>>>>>>>>> /etc/ssh/ssh_host_ed25519_key.pub
>>>>>>>>>>>> /etc/ssh/ssh_host_rsa_key
>>>>>>>>>>>> /etc/ssh/ssh_host_rsa_key.pub
>>>>>>>>>>>> //etc/sudoers
>>>>>>>>>>>> /etc/sysconfig/createfiles
>>>>>>>>>>>> /etc/sysconfig/firewall.local
>>>>>>>>>>>> /etc/sysconfig/lm_sensors
>>>>>>>>>>>> /etc/sysconfig/modules
>>>>>>>>>>>> /etc/sysconfig/ramdisk
>>>>>>>>>>>> /etc/sysconfig/rc
>>>>>>>>>>>> /etc/sysconfig/rc.local
>>>>>>>>>>>> /etc/unbound
>>>>>>>>>>>> //home/ahb
>>>>>>>>>>>> /root/.bash_history
>>>>>>>>>>>> /root/.gitconfig
>>>>>>>>>>>> /root/.ssh
>>>>>>>>>>>> /var/ipfire/accounting/settings.conf
>>>>>>>>>>>> /var/ipfire/auth/users
>>>>>>>>>>>> /var/ipfire/backup/addons/backup
>>>>>>>>>>>> /var/ipfire/backup/exclude.user
>>>>>>>>>>>> /var/ipfire/backup/include.user
>>>>>>>>>>>> /var/ipfire/ca/cacert.pem
>>>>>>>>>>>> /var/ipfire/captive/agb.txt
>>>>>>>>>>>> /var/ipfire/captive/clients
>>>>>>>>>>>> /var/ipfire/captive/coupons
>>>>>>>>>>>> /var/ipfire/captive/logo.dat
>>>>>>>>>>>> /var/ipfire/captive/settings
>>>>>>>>>>>> /var/ipfire/captive/terms.txt
>>>>>>>>>>>> /var/ipfire/captive/voucher_out
>>>>>>>>>>>> /var/ipfire/certs/hostcert.pem
>>>>>>>>>>>> /var/ipfire/certs/hostkey.pem
>>>>>>>>>>>> /var/ipfire/certs/phoebevmipseccert.pem
>>>>>>>>>>>> /var/ipfire/connscheduler/connscheduler.conf
>>>>>>>>>>>> /var/ipfire/crls/cacrl.pem
>>>>>>>>>>>> /var/ipfire/cups/cups-browsed.conf
>>>>>>>>>>>> /var/ipfire/cups/subscriptions.conf
>>>>>>>>>>>> /var/ipfire/ddns/config
>>>>>>>>>>>> /var/ipfire/ddns/ddns.conf
>>>>>>>>>>>> /var/ipfire/ddns/settings
>>>>>>>>>>>> /var/ipfire/dhcp/advoptions
>>>>>>>>>>>> /var/ipfire/dhcp/advoptions-list
>>>>>>>>>>>> /var/ipfire/dhcpc/dhcpcd.conf
>>>>>>>>>>>> /var/ipfire/dhcp/dhcpd.conf
>>>>>>>>>>>> /var/ipfire/dhcp/dhcpd.conf.local
>>>>>>>>>>>> /var/ipfire/dhcp/enable_blue
>>>>>>>>>>>> /var/ipfire/dhcp/enable_green
>>>>>>>>>>>> /var/ipfire/dhcp/fixleases
>>>>>>>>>>>> /var/ipfire/dhcp/settings
>>>>>>>>>>>> /var/ipfire/dma/auth.conf
>>>>>>>>>>>> /var/ipfire/dma/dma.conf
>>>>>>>>>>>> /var/ipfire/dma/mail.conf
>>>>>>>>>>>> /var/ipfire/dns
>>>>>>>>>>>> /var/ipfire/dnsforward/config
>>>>>>>>>>>> /var/ipfire/dns/settings
>>>>>>>>>>>> /var/ipfire/ethernet/aliases
>>>>>>>>>>>> /var/ipfire/ethernet/settings
>>>>>>>>>>>> /var/ipfire/ethernet/wireless
>>>>>>>>>>>> /var/ipfire/extrahd/settings
>>>>>>>>>>>> /var/ipfire/firewall
>>>>>>>>>>>> /var/ipfire/firewall/config
>>>>>>>>>>>> /var/ipfire/firewall/settings
>>>>>>>>>>>> /var/ipfire/fwhosts
>>>>>>>>>>>> /var/ipfire/isdn/settings
>>>>>>>>>>>> /var/ipfire/logging/settings
>>>>>>>>>>>> /var/ipfire/mac/settings
>>>>>>>>>>>> /var/ipfire/main/firstsetup_ok
>>>>>>>>>>>> /var/ipfire/main/gpl_accepted
>>>>>>>>>>>> /var/ipfire/main/hostname.conf
>>>>>>>>>>>> /var/ipfire/main/hosts
>>>>>>>>>>>> /var/ipfire/main/manualpages
>>>>>>>>>>>> /var/ipfire/main/routing
>>>>>>>>>>>> /var/ipfire/main/security
>>>>>>>>>>>> /var/ipfire/main/send_profile
>>>>>>>>>>>> /var/ipfire/main/settings
>>>>>>>>>>>> /var/ipfire/modem/settings
>>>>>>>>>>>> /var/ipfire/optionsfw/settings
>>>>>>>>>>>> /var/ipfire/ovpn
>>>>>>>>>>>> /var/ipfire/ovpn/ccd.conf
>>>>>>>>>>>> /var/ipfire/ovpn/collectd.vpn
>>>>>>>>>>>> /var/ipfire/ovpn/enable
>>>>>>>>>>>> /var/ipfire/ovpn/server.conf
>>>>>>>>>>>> /var/ipfire/ovpn/settings
>>>>>>>>>>>> /var/ipfire/pakfire/settings
>>>>>>>>>>>> /var/ipfire/ppp
>>>>>>>>>>>> /var/ipfire/ppp/fake-resolv.conf
>>>>>>>>>>>> /var/ipfire/ppp/settings
>>>>>>>>>>>> /var/ipfire/private/cakey.pem
>>>>>>>>>>>> /var/ipfire/proxy
>>>>>>>>>>>> /var/ipfire/proxy/asnbl-helper.conf
>>>>>>>>>>>> /var/ipfire/proxy/cachemgr.conf
>>>>>>>>>>>> /var/ipfire/proxy/enable
>>>>>>>>>>>> /var/ipfire/proxy/settings
>>>>>>>>>>>> /var/ipfire/proxy/squid.conf
>>>>>>>>>>>> /var/ipfire/qos/bin
>>>>>>>>>>>> /var/ipfire/qos/bin/qos.sh
>>>>>>>>>>>> /var/ipfire/qos/classes
>>>>>>>>>>>> /var/ipfire/qos/level7config
>>>>>>>>>>>> /var/ipfire/qos/portconfig
>>>>>>>>>>>> /var/ipfire/qos/settings
>>>>>>>>>>>> /var/ipfire/qos/subclasses
>>>>>>>>>>>> /var/ipfire/qos/tosconfig
>>>>>>>>>>>> /var/ipfire/remote/enablessh
>>>>>>>>>>>> /var/ipfire/remote/settings
>>>>>>>>>>>> /var/ipfire/sensors/settings
>>>>>>>>>>>> /var/ipfire/suricata/oinkmaster.conf
>>>>>>>>>>>> /var/ipfire/suricata/oinkmaster-modify-sids.conf
>>>>>>>>>>>> /var/ipfire/suricata/oinkmaster-provider-includes.conf
>>>>>>>>>>>> /var/ipfire/suricata/providers-settings
>>>>>>>>>>>> /var/ipfire/suricata/settings
>>>>>>>>>>>> /var/ipfire/suricata/suricata-default-rules.yaml
>>>>>>>>>>>> /var/ipfire/suricata/suricata-dns-servers.yaml
>>>>>>>>>>>> /var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
>>>>>>>>>>>> /var/ipfire/suricata/suricata-homenet.yaml
>>>>>>>>>>>> /var/ipfire/suricata/suricata-http-ports.yaml
>>>>>>>>>>>> /var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml
>>>>>>>>>>>> /var/ipfire/suricata/suricata-used-providers.yaml
>>>>>>>>>>>> /var/ipfire/time/
>>>>>>>>>>>> /var/ipfire/time/counter.conf
>>>>>>>>>>>> /var/ipfire/time/enable
>>>>>>>>>>>> /var/ipfire/time/settime.conf
>>>>>>>>>>>> /var/ipfire/time/settings
>>>>>>>>>>>> /var/ipfire/upnp/settings
>>>>>>>>>>>> /var/ipfire/urlfilter
>>>>>>>>>>>> /var/ipfire/urlfilter/settings
>>>>>>>>>>>> /var/ipfire/urlfilter/squidGuard.conf
>>>>>>>>>>>> /var/ipfire/vpn
>>>>>>>>>>>> /var/ipfire/vpn/config
>>>>>>>>>>>> /var/ipfire/vpn/ipsec.conf
>>>>>>>>>>>> /var/ipfire/vpn/settings
>>>>>>>>>>>> /var/ipfire/wakeonlan/clients.conf
>>>>>>>>>>>> /var/ipfire/wio/wio.conf
>>>>>>>>>>>> /var/ipfire/wireless/config
>>>>>>>>>>>> /var/ipfire/wireless/settings
>>>>>>>>>>>> /var/lib/suricata
>>>>>>>>>>>> /var/log/rrd/collectd
>>>>>>>>>>>> /var/log/rrd/hddshutdown-md127.rrd
>>>>>>>>>>>> /var/log/rrd/hddshutdown-sda.rrd
>>>>>>>>>>>> /var/log/rrd/hddshutdown-sdb.rrd
>>>>>>>>>>>> /var/log/rrd/hddtemp-md127.rrd
>>>>>>>>>>>> /var/log/rrd/hddtemp-sda.rrd
>>>>>>>>>>>> /var/log/rrd/hddtemp-sdb.rrd
>>>>>>>>>>>> /var/log/rrd/wio
>>>>>>>>>>>> /var/log/vnstat
>>>>>>>>>>>> /var/tmp/idsrules-emerging.tar.gz
>>>>>>>>>>>> /var/tmp/idsrules-sslbl_blacklist.rules
>>>>>>>>>>>> 
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Bernhard
>>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On 29 Mar 2022, at 14:10, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Just tried this patch out on my vm testbed system and it still doesn't work for me. The backup file had got to 1.3GB when I deleted the backup file as it was still growing. The normal correct backup file on that vm machine is around 7MB
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> The above was the case for both running it from the WUI or from the command line from my unprivileged user using sudo backupctrl exclude
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On 29/03/2022 14:27, Michael Tremer wrote:
>>>>>>>>>>>>>>>> This patch fixes globbing expansion in the backup include file list
>>>>>>>>>>>>>>>> which got broken in c7e0d73e7cfd7be95db9d0a5f3392b8241813d5b.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>> config/backup/backup.pl | 8 ++++----
>>>>>>>>>>>>>>>> 1 file changed, 4 insertions(+), 4 deletions(-)
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> diff --git a/config/backup/backup.pl b/config/backup/backup.pl
>>>>>>>>>>>>>>>> index a2337cf23..6f9295e94 100644
>>>>>>>>>>>>>>>> --- a/config/backup/backup.pl
>>>>>>>>>>>>>>>> +++ b/config/backup/backup.pl
>>>>>>>>>>>>>>>> @@ -19,6 +19,8 @@
>>>>>>>>>>>>>>>> # #
>>>>>>>>>>>>>>>> ###############################################################################
>>>>>>>>>>>>>>>> +shopt -s nullglob
>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>> NOW="$(date "+%Y-%m-%d-%H:%M")"
>>>>>>>>>>>>>>>> list_addons() {
>>>>>>>>>>>>>>>> @@ -38,10 +40,8 @@ process_includes() {
>>>>>>>>>>>>>>>> for include in $@; do
>>>>>>>>>>>>>>>> local file
>>>>>>>>>>>>>>>> while read -r file; do
>>>>>>>>>>>>>>>> -  for file in ${file}; do
>>>>>>>>>>>>>>>> -  if [ -e "/${file}" ]; then
>>>>>>>>>>>>>>>> -  echo "${file}"
>>>>>>>>>>>>>>>> -  fi
>>>>>>>>>>>>>>>> +  for file in /${file}; do
>>>>>>>>>>>>>>>> +  echo "${file}"
>>>>>>>>>>>>>>>> done
>>>>>>>>>>>>>>>> done < "${include}"
>>>>>>>>>>>>>>>> done | sort -u

Bernhard Bitsch March 31, 2022, 10:02 a.m. UTC | #21

Hi all,

yesterday I did some debugging and verification of the function:
- if you only use the list functionality of backup.pl, you can insert 
echo commands to see the processing
- including the root directory for blank lines can be explained looking 
at this debug output. A blank line gives a file name of length 0, 
prepending '/' results in '/', which exists.
- in testing I had a little problem ( which we should keep in mind in 
analysing problems from the community ); for a quick work flow, I just 
put the test cases into include.user ( edited on my Windows computer ). 
But I forgot to check the line ending. So empty lines gave file names of 
length 1 ( the CR ).


Some theoretical thoughts.
Having a background in theoretic informatics and compiler construction 
the recursive solution with pushd/popd was the most quick and elegant 
solution for me. On the other hand it is known, that many recursive 
solutions have an iterative equivalent. For our problem this is Michaels 
solution. My checks show that both solutions are equivalent, with the 
check for empty strings.


Regards,
Bernhard

Am 31.03.2022 um 09:46 schrieb Michael Tremer:
> Hello Adolf,
> 
> Brilliant. Thank you very much for testing this in detail.
> 
> Best,
> -Michael
> 
>> On 31 Mar 2022, at 08:41, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>
>> Hi Michael,
>>
>> On 30/03/2022 23:20, Adolf Belka wrote:
>>> Hi,
>>>
>>> On 30/03/2022 17:05, Michael Tremer wrote:
>>>> Hello Bernhard,
>>>>
>>>> You are obviously right. My brain is absolute mush these days and doesn’t allow me to function.
>>>>
>>>> I will correct this by the end of the day.
>>>>
>>>> -Michael
>>>>
>>>>> On 30 Mar 2022, at 15:38, Bernhard Bitsch <bbitsch@ipfire.org> wrote:
>>>>>
>>>>> Hello all,
>>>>>
>>>>>
>>>>> Am 30.03.2022 um 16:06 schrieb Michael Tremer:
>>>>>> Hello everyone,
>>>>>> Thank you very much for helping me test this.
>>>>>> I decided that we will have to create an emergency core update for this, because it will otherwise break people’s system. We also have a vulnerability in zlib which is being handed around by the press, so that has helped me to make a decision.
>>>>>
>>>>> I think this is a good solution. Shall we add a warning to backup wiki page? Because in the moment a backup -> reinstall -> restore process leaves an incomplete system.
>>>>>
>>>>>> I also pushed two more patches. One filters out any empty lines because they have quite a damaging effect and it happens easily that they are added to the include list. Secondly, I brought back the existence check which I thought was only there for the globbing mechanism, but also had the small side-effect of filtering out the file list early on which helps keeping tar happy.
>>>>>> Please review those changes and please install the new c166 from testing as soon as the build has finished (I am not sure if you are all on the nightly-builds list to see any notifications).
>>> I am not on the notifications list but I will check first thing tomorrow and when the latest has been updated I will do a test install and evaluation of the backup and provide confirmation of how it functions.
>>>
>> As CU166 has been moved to master I took a clone of my CU165 vm and then did a CU on Testing with it rather than doing a fresh install. If you would also want a test of a fresh install done I can also do that.
>>
>> The version I tested was IPFire 2.27 (x86_64) - Core Update 166 Development Build: master/8f696f60
>>
>> Anyway the backup is working fine again. All the directories are present. The size of the backup is now very similar to previously also confirming all contents should be good.
>>
>> I also added back in a couple of blank lines into my include.user file and the backup worked without any problems.
>>
>> I also did a restore of the backup and again as far as I can tell with a quick check through on my testbed, everything is fine.
>>
>> So everything looks good to me.
>>
>> Regards,
>>
>> Adolf.
>>
>>> Regards,
>>> Adolf.
>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=4f0e7f24f293ca10096e616010cfd4417afc157c
>>>>> The test on empty string should be in the 'while read -r file' loop. The empty lines are in the include files. The outer for loop iterates over the file list.
>>>>>
>>>>> Regards,
>>>>> Bernhard
>>>>>
>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=b275771fdd10ac7116e5d400dc10d3148df8ac27
>>>>>> Best,
>>>>>> -Michael
>>>>>>> On 29 Mar 2022, at 20:21, Bernhard Bitsch <bbitsch@ipfire.org> wrote:
>>>>>>>
>>>>>>> Just checked the blank line problem.
>>>>>>> Pattern is empty , file name list is '/' --> no names produced ( with existence check )
>>>>>>> Pattern is empty --> produced file name '/' ( without -e check )
>>>>>>>
>>>>>>> Am 29.03.2022 um 20:36 schrieb Bernhard Bitsch:
>>>>>>>> Hi all,
>>>>>>>> Am 29.03.2022 um 17:23 schrieb Adolf Belka:
>>>>>>>>> Hi All,
>>>>>>>>>
>>>>>>>>> On 29/03/2022 16:22, Michael Tremer wrote:
>>>>>>>>>>
>>>>>>>>>>> On 29 Mar 2022, at 15:22, Bernhard Bitsch <bbitsch@ipfire.org> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Looks like your include.user file contains absolute paths.
>>>>>>>>> That was not the problem but I have corrected that. I would have expected that when the main include and exclude files were made relative that the update script would also have checked for any entries in the include.user and exclude.user files but it looks like it didn't.th
>>>>>>>> According to the source they are chosen.
>>>>>>>>>>
>>>>>>>>>> Or an empty line maybe?
>>>>>>>>> That turned out to be the problem. There was an empty line after the two lines I had added. You only saw it if you scrolled the pointer down the file in the editor.
>>>>>>>> Interesting. A blank line should produce nothing.
>>>>>>>>>
>>>>>>>>> Removing that blank line removed the line only with a / and the backup then successfully ran and created a backup file containing all the directories expected.
>>>>>>>>>
>>>>>>>>> The only thing I found is that there was the following tar error message
>>>>>>>>>
>>>>>>>>> tar: Exiting with failure status due to previous errors
>>>>>>>>>
>>>>>>>>> The only thing I found in the backup output was several lines such as
>>>>>>>>>
>>>>>>>>> tar: /etc/squid/squid.conf.local: Cannot stat: No such file or directory
>>>>>>>>> tar: /etc/squid/squid.conf.pre.local: Cannot stat: No such file or directory
>>>>>>>>>
>>>>>>>>> There were seven lines in total like this and the three I checked were files that did not exist on my system. I presume that these are files that should be backed up if present but don't have to be present so that the error messages are nothing to worry about.
>>>>>>>>> There were no error messages when doing the backup via the WUI.
>>>>>>>>>
>>>>>>>> Good find. ;)
>>>>>>>> Michael's patch removes the test of existence, globbing produces existent file names only. But I suppose there are several full names of files, not existing in each implementation.
>>>>>>>> Conclusion: Repair the file globbing by the shopt and do the existence check. So all existent files are chosen, that are listed in the include files.
>>>>>>>> Regards,
>>>>>>>> Bernhard
>>>>>>>>> The blank line was only on my vm testbed system but as I clone this for any testing activities I will have to check all of my existing vm's to fix this.
>>>>>>>>>
>>>>>>>>> Glad we resolved this simply. Sorry for the anxiety.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Adolf.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Am 29.03.2022 um 16:14 schrieb Adolf Belka:
>>>>>>>>>>>> Hi All,
>>>>>>>>>>>> This is what I get with the pushd/popd version in place on the same system
>>>>>>>>>>>> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
>>>>>>>>>>>> etc/group
>>>>>>>>>>>> etc/hosts
>>>>>>>>>>>> etc/hosts.allow
>>>>>>>>>>>> etc/hosts.deny
>>>>>>>>>>>> etc/httpd/server.crt
>>>>>>>>>>>> etc/httpd/server.csr
>>>>>>>>>>>> etc/httpd/server-ecdsa.crt
>>>>>>>>>>>> etc/httpd/server-ecdsa.csr
>>>>>>>>>>>> etc/httpd/server-ecdsa.key
>>>>>>>>>>>> etc/httpd/server.key
>>>>>>>>>>>> etc/ipsec.user.conf
>>>>>>>>>>>> etc/ipsec.user.secrets
>>>>>>>>>>>> etc/logrotate.d
>>>>>>>>>>>> etc/passwd
>>>>>>>>>>>> etc/shadow
>>>>>>>>>>>> etc/ssh/sshd_config
>>>>>>>>>>>> etc/ssh/ssh_host_ecdsa_key
>>>>>>>>>>>> etc/ssh/ssh_host_ecdsa_key.pub
>>>>>>>>>>>> etc/ssh/ssh_host_ed25519_key
>>>>>>>>>>>> etc/ssh/ssh_host_ed25519_key.pub
>>>>>>>>>>>> etc/ssh/ssh_host_rsa_key
>>>>>>>>>>>> etc/ssh/ssh_host_rsa_key.pub
>>>>>>>>>>>> /etc/sudoers
>>>>>>>>>>>> etc/sysconfig/createfiles
>>>>>>>>>>>> etc/sysconfig/firewall.local
>>>>>>>>>>>> etc/sysconfig/lm_sensors
>>>>>>>>>>>> etc/sysconfig/modules
>>>>>>>>>>>> etc/sysconfig/ramdisk
>>>>>>>>>>>> etc/sysconfig/rc
>>>>>>>>>>>> etc/sysconfig/rc.local
>>>>>>>>>>>> etc/unbound
>>>>>>>>>>>> /home/ahb
>>>>>>>>>>>> root/.bash_history
>>>>>>>>>>>> var/ipfire/accounting/settings.conf
>>>>>>>>>>>> var/ipfire/auth/users
>>>>>>>>>>>> var/ipfire/backup/addons/backup
>>>>>>>>>>>> var/ipfire/backup/exclude.user
>>>>>>>>>>>> var/ipfire/backup/include.user
>>>>>>>>>>>> var/ipfire/ca/cacert.pem
>>>>>>>>>>>> var/ipfire/captive/agb.txt
>>>>>>>>>>>> var/ipfire/captive/clients
>>>>>>>>>>>> var/ipfire/captive/coupons
>>>>>>>>>>>> var/ipfire/captive/logo.dat
>>>>>>>>>>>> var/ipfire/captive/settings
>>>>>>>>>>>> var/ipfire/captive/terms.txt
>>>>>>>>>>>> var/ipfire/captive/voucher_out
>>>>>>>>>>>> var/ipfire/certs/hostcert.pem
>>>>>>>>>>>> var/ipfire/certs/hostkey.pem
>>>>>>>>>>>> var/ipfire/certs/phoebevmipseccert.pem
>>>>>>>>>>>> var/ipfire/connscheduler/connscheduler.conf
>>>>>>>>>>>> var/ipfire/crls/cacrl.pem
>>>>>>>>>>>> var/ipfire/cups/cups-browsed.conf
>>>>>>>>>>>> var/ipfire/cups/subscriptions.conf
>>>>>>>>>>>> var/ipfire/ddns/config
>>>>>>>>>>>> var/ipfire/ddns/ddns.conf
>>>>>>>>>>>> var/ipfire/ddns/settings
>>>>>>>>>>>> var/ipfire/dhcp/advoptions
>>>>>>>>>>>> var/ipfire/dhcp/advoptions-list
>>>>>>>>>>>> var/ipfire/dhcpc/dhcpcd.conf
>>>>>>>>>>>> var/ipfire/dhcp/dhcpd.conf
>>>>>>>>>>>> var/ipfire/dhcp/dhcpd.conf.local
>>>>>>>>>>>> var/ipfire/dhcp/enable_blue
>>>>>>>>>>>> var/ipfire/dhcp/enable_green
>>>>>>>>>>>> var/ipfire/dhcp/fixleases
>>>>>>>>>>>> var/ipfire/dhcp/settings
>>>>>>>>>>>> var/ipfire/dma/auth.conf
>>>>>>>>>>>> var/ipfire/dma/dma.conf
>>>>>>>>>>>> var/ipfire/dma/mail.conf
>>>>>>>>>>>> var/ipfire/dns
>>>>>>>>>>>> var/ipfire/dnsforward/config
>>>>>>>>>>>> var/ipfire/dns/settings
>>>>>>>>>>>> var/ipfire/ethernet/aliases
>>>>>>>>>>>> var/ipfire/ethernet/settings
>>>>>>>>>>>> var/ipfire/ethernet/wireless
>>>>>>>>>>>> var/ipfire/extrahd/settings
>>>>>>>>>>>> var/ipfire/firewall
>>>>>>>>>>>> var/ipfire/firewall/config
>>>>>>>>>>>> var/ipfire/firewall/settings
>>>>>>>>>>>> var/ipfire/fwhosts
>>>>>>>>>>>> var/ipfire/isdn/settings
>>>>>>>>>>>> var/ipfire/logging/settings
>>>>>>>>>>>> var/ipfire/mac/settings
>>>>>>>>>>>> var/ipfire/main/firstsetup_ok
>>>>>>>>>>>> var/ipfire/main/gpl_accepted
>>>>>>>>>>>> var/ipfire/main/hostname.conf
>>>>>>>>>>>> var/ipfire/main/hosts
>>>>>>>>>>>> var/ipfire/main/manualpages
>>>>>>>>>>>> var/ipfire/main/routing
>>>>>>>>>>>> var/ipfire/main/security
>>>>>>>>>>>> var/ipfire/main/send_profile
>>>>>>>>>>>> var/ipfire/main/settings
>>>>>>>>>>>> var/ipfire/modem/settings
>>>>>>>>>>>> var/ipfire/optionsfw/settings
>>>>>>>>>>>> var/ipfire/ovpn
>>>>>>>>>>>> var/ipfire/ovpn/ccd.conf
>>>>>>>>>>>> var/ipfire/ovpn/collectd.vpn
>>>>>>>>>>>> var/ipfire/ovpn/enable
>>>>>>>>>>>> var/ipfire/ovpn/server.conf
>>>>>>>>>>>> var/ipfire/ovpn/settings
>>>>>>>>>>>> var/ipfire/pakfire/settings
>>>>>>>>>>>> var/ipfire/ppp
>>>>>>>>>>>> var/ipfire/ppp/fake-resolv.conf
>>>>>>>>>>>> var/ipfire/ppp/settings
>>>>>>>>>>>> var/ipfire/private/cakey.pem
>>>>>>>>>>>> var/ipfire/proxy
>>>>>>>>>>>> var/ipfire/proxy/asnbl-helper.conf
>>>>>>>>>>>> var/ipfire/proxy/cachemgr.conf
>>>>>>>>>>>> var/ipfire/proxy/enable
>>>>>>>>>>>> var/ipfire/proxy/settings
>>>>>>>>>>>> var/ipfire/proxy/squid.conf
>>>>>>>>>>>> var/ipfire/qos/bin
>>>>>>>>>>>> var/ipfire/qos/classes
>>>>>>>>>>>> var/ipfire/qos/level7config
>>>>>>>>>>>> var/ipfire/qos/portconfig
>>>>>>>>>>>> var/ipfire/qos/settings
>>>>>>>>>>>> var/ipfire/qos/subclasses
>>>>>>>>>>>> var/ipfire/qos/tosconfig
>>>>>>>>>>>> var/ipfire/remote/enablessh
>>>>>>>>>>>> var/ipfire/remote/settings
>>>>>>>>>>>> var/ipfire/sensors/settings
>>>>>>>>>>>> var/ipfire/suricata/oinkmaster.conf
>>>>>>>>>>>> var/ipfire/suricata/oinkmaster-modify-sids.conf
>>>>>>>>>>>> var/ipfire/suricata/oinkmaster-provider-includes.conf
>>>>>>>>>>>> var/ipfire/suricata/providers-settings
>>>>>>>>>>>> var/ipfire/suricata/settings
>>>>>>>>>>>> var/ipfire/suricata/suricata-default-rules.yaml
>>>>>>>>>>>> var/ipfire/suricata/suricata-dns-servers.yaml
>>>>>>>>>>>> var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
>>>>>>>>>>>> var/ipfire/suricata/suricata-homenet.yaml
>>>>>>>>>>>> var/ipfire/suricata/suricata-http-ports.yaml
>>>>>>>>>>>> var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml
>>>>>>>>>>>> var/ipfire/suricata/suricata-used-providers.yaml
>>>>>>>>>>>> var/ipfire/time/
>>>>>>>>>>>> var/ipfire/time/counter.conf
>>>>>>>>>>>> var/ipfire/time/enable
>>>>>>>>>>>> var/ipfire/time/settime.conf
>>>>>>>>>>>> var/ipfire/time/settings
>>>>>>>>>>>> var/ipfire/upnp/settings
>>>>>>>>>>>> var/ipfire/urlfilter
>>>>>>>>>>>> var/ipfire/urlfilter/settings
>>>>>>>>>>>> var/ipfire/urlfilter/squidGuard.conf
>>>>>>>>>>>> var/ipfire/vpn
>>>>>>>>>>>> var/ipfire/vpn/config
>>>>>>>>>>>> var/ipfire/vpn/ipsec.conf
>>>>>>>>>>>> var/ipfire/vpn/settings
>>>>>>>>>>>> var/ipfire/wakeonlan/clients.conf
>>>>>>>>>>>> var/ipfire/wio/wio.conf
>>>>>>>>>>>> var/ipfire/wireless/config
>>>>>>>>>>>> var/ipfire/wireless/settings
>>>>>>>>>>>> var/lib/suricata
>>>>>>>>>>>> var/log/rrd/collectd
>>>>>>>>>>>> var/log/rrd/hddshutdown-md127.rrd
>>>>>>>>>>>> var/log/rrd/hddshutdown-sda.rrd
>>>>>>>>>>>> var/log/rrd/hddshutdown-sdb.rrd
>>>>>>>>>>>> var/log/rrd/hddtemp-md127.rrd
>>>>>>>>>>>> var/log/rrd/hddtemp-sda.rrd
>>>>>>>>>>>> var/log/rrd/hddtemp-sdb.rrd
>>>>>>>>>>>> var/log/rrd/wio
>>>>>>>>>>>> var/log/vnstat
>>>>>>>>>>>> var/tmp/idsrules-emerging.tar.gz
>>>>>>>>>>>> var/tmp/idsrules-sslbl_blacklist.rules
>>>>>>>>>>>> The following are in the previous list but not in this one:-
>>>>>>>>>>>> /
>>>>>>>>>>>> /etc/conntrackd/conntrackd.conf
>>>>>>>>>>>> /etc/ipsec.user-post.conf
>>>>>>>>>>>> /root/.gitconfig
>>>>>>>>>>>> /root/.ssh
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Adolf.
>>>>>>>>>>>> On 29/03/2022 16:01, Adolf Belka wrote:
>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 29/03/2022 15:36, Bernhard Bitsch wrote:
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Am 29.03.2022 um 15:11 schrieb Michael Tremer:
>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Could you please send the file listing to find out what is being included what shouldn’t?
>>>>>>>>>>>>> Unfortunately, as I stopped the backup continuing once it had reached 1.2GB, the file created was not able to be opened. Probably stopping the backup corrupted it in some way.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Just for easy quick handling, you can use
>>>>>>>>>>>>>> '/var/ipfire/backup/bin/backup.pl list' to show the files included.
>>>>>>>>>>>>> I gave this a go and here is the output from the command.
>>>>>>>>>>>>>
>>>>>>>>>>>>> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
>>>>>>>>>>>>> /
>>>>>>>>>>>>> /etc/conntrackd/conntrackd.conf
>>>>>>>>>>>>> /etc/group
>>>>>>>>>>>>> /etc/hosts
>>>>>>>>>>>>> /etc/hosts.allow
>>>>>>>>>>>>> /etc/hosts.deny
>>>>>>>>>>>>> /etc/httpd/server.crt
>>>>>>>>>>>>> /etc/httpd/server.csr
>>>>>>>>>>>>> /etc/httpd/server-ecdsa.crt
>>>>>>>>>>>>> /etc/httpd/server-ecdsa.csr
>>>>>>>>>>>>> /etc/httpd/server-ecdsa.key
>>>>>>>>>>>>> /etc/httpd/server.key
>>>>>>>>>>>>> /etc/ipsec.user.conf
>>>>>>>>>>>>> /etc/ipsec.user-post.conf
>>>>>>>>>>>>> /etc/ipsec.user.secrets
>>>>>>>>>>>>> /etc/logrotate.d
>>>>>>>>>>>>> /etc/passwd
>>>>>>>>>>>>> /etc/shadow
>>>>>>>>>>>>> /etc/squid/squid.conf.local
>>>>>>>>>>>>> /etc/squid/squid.conf.pre.local
>>>>>>>>>>>>> /etc/ssh/sshd_config
>>>>>>>>>>>>> /etc/ssh/ssh_host_ecdsa_key
>>>>>>>>>>>>> /etc/ssh/ssh_host_ecdsa_key.pub
>>>>>>>>>>>>> /etc/ssh/ssh_host_ed25519_key
>>>>>>>>>>>>> /etc/ssh/ssh_host_ed25519_key.pub
>>>>>>>>>>>>> /etc/ssh/ssh_host_rsa_key
>>>>>>>>>>>>> /etc/ssh/ssh_host_rsa_key.pub
>>>>>>>>>>>>> //etc/sudoers
>>>>>>>>>>>>> /etc/sysconfig/createfiles
>>>>>>>>>>>>> /etc/sysconfig/firewall.local
>>>>>>>>>>>>> /etc/sysconfig/lm_sensors
>>>>>>>>>>>>> /etc/sysconfig/modules
>>>>>>>>>>>>> /etc/sysconfig/ramdisk
>>>>>>>>>>>>> /etc/sysconfig/rc
>>>>>>>>>>>>> /etc/sysconfig/rc.local
>>>>>>>>>>>>> /etc/unbound
>>>>>>>>>>>>> //home/ahb
>>>>>>>>>>>>> /root/.bash_history
>>>>>>>>>>>>> /root/.gitconfig
>>>>>>>>>>>>> /root/.ssh
>>>>>>>>>>>>> /var/ipfire/accounting/settings.conf
>>>>>>>>>>>>> /var/ipfire/auth/users
>>>>>>>>>>>>> /var/ipfire/backup/addons/backup
>>>>>>>>>>>>> /var/ipfire/backup/exclude.user
>>>>>>>>>>>>> /var/ipfire/backup/include.user
>>>>>>>>>>>>> /var/ipfire/ca/cacert.pem
>>>>>>>>>>>>> /var/ipfire/captive/agb.txt
>>>>>>>>>>>>> /var/ipfire/captive/clients
>>>>>>>>>>>>> /var/ipfire/captive/coupons
>>>>>>>>>>>>> /var/ipfire/captive/logo.dat
>>>>>>>>>>>>> /var/ipfire/captive/settings
>>>>>>>>>>>>> /var/ipfire/captive/terms.txt
>>>>>>>>>>>>> /var/ipfire/captive/voucher_out
>>>>>>>>>>>>> /var/ipfire/certs/hostcert.pem
>>>>>>>>>>>>> /var/ipfire/certs/hostkey.pem
>>>>>>>>>>>>> /var/ipfire/certs/phoebevmipseccert.pem
>>>>>>>>>>>>> /var/ipfire/connscheduler/connscheduler.conf
>>>>>>>>>>>>> /var/ipfire/crls/cacrl.pem
>>>>>>>>>>>>> /var/ipfire/cups/cups-browsed.conf
>>>>>>>>>>>>> /var/ipfire/cups/subscriptions.conf
>>>>>>>>>>>>> /var/ipfire/ddns/config
>>>>>>>>>>>>> /var/ipfire/ddns/ddns.conf
>>>>>>>>>>>>> /var/ipfire/ddns/settings
>>>>>>>>>>>>> /var/ipfire/dhcp/advoptions
>>>>>>>>>>>>> /var/ipfire/dhcp/advoptions-list
>>>>>>>>>>>>> /var/ipfire/dhcpc/dhcpcd.conf
>>>>>>>>>>>>> /var/ipfire/dhcp/dhcpd.conf
>>>>>>>>>>>>> /var/ipfire/dhcp/dhcpd.conf.local
>>>>>>>>>>>>> /var/ipfire/dhcp/enable_blue
>>>>>>>>>>>>> /var/ipfire/dhcp/enable_green
>>>>>>>>>>>>> /var/ipfire/dhcp/fixleases
>>>>>>>>>>>>> /var/ipfire/dhcp/settings
>>>>>>>>>>>>> /var/ipfire/dma/auth.conf
>>>>>>>>>>>>> /var/ipfire/dma/dma.conf
>>>>>>>>>>>>> /var/ipfire/dma/mail.conf
>>>>>>>>>>>>> /var/ipfire/dns
>>>>>>>>>>>>> /var/ipfire/dnsforward/config
>>>>>>>>>>>>> /var/ipfire/dns/settings
>>>>>>>>>>>>> /var/ipfire/ethernet/aliases
>>>>>>>>>>>>> /var/ipfire/ethernet/settings
>>>>>>>>>>>>> /var/ipfire/ethernet/wireless
>>>>>>>>>>>>> /var/ipfire/extrahd/settings
>>>>>>>>>>>>> /var/ipfire/firewall
>>>>>>>>>>>>> /var/ipfire/firewall/config
>>>>>>>>>>>>> /var/ipfire/firewall/settings
>>>>>>>>>>>>> /var/ipfire/fwhosts
>>>>>>>>>>>>> /var/ipfire/isdn/settings
>>>>>>>>>>>>> /var/ipfire/logging/settings
>>>>>>>>>>>>> /var/ipfire/mac/settings
>>>>>>>>>>>>> /var/ipfire/main/firstsetup_ok
>>>>>>>>>>>>> /var/ipfire/main/gpl_accepted
>>>>>>>>>>>>> /var/ipfire/main/hostname.conf
>>>>>>>>>>>>> /var/ipfire/main/hosts
>>>>>>>>>>>>> /var/ipfire/main/manualpages
>>>>>>>>>>>>> /var/ipfire/main/routing
>>>>>>>>>>>>> /var/ipfire/main/security
>>>>>>>>>>>>> /var/ipfire/main/send_profile
>>>>>>>>>>>>> /var/ipfire/main/settings
>>>>>>>>>>>>> /var/ipfire/modem/settings
>>>>>>>>>>>>> /var/ipfire/optionsfw/settings
>>>>>>>>>>>>> /var/ipfire/ovpn
>>>>>>>>>>>>> /var/ipfire/ovpn/ccd.conf
>>>>>>>>>>>>> /var/ipfire/ovpn/collectd.vpn
>>>>>>>>>>>>> /var/ipfire/ovpn/enable
>>>>>>>>>>>>> /var/ipfire/ovpn/server.conf
>>>>>>>>>>>>> /var/ipfire/ovpn/settings
>>>>>>>>>>>>> /var/ipfire/pakfire/settings
>>>>>>>>>>>>> /var/ipfire/ppp
>>>>>>>>>>>>> /var/ipfire/ppp/fake-resolv.conf
>>>>>>>>>>>>> /var/ipfire/ppp/settings
>>>>>>>>>>>>> /var/ipfire/private/cakey.pem
>>>>>>>>>>>>> /var/ipfire/proxy
>>>>>>>>>>>>> /var/ipfire/proxy/asnbl-helper.conf
>>>>>>>>>>>>> /var/ipfire/proxy/cachemgr.conf
>>>>>>>>>>>>> /var/ipfire/proxy/enable
>>>>>>>>>>>>> /var/ipfire/proxy/settings
>>>>>>>>>>>>> /var/ipfire/proxy/squid.conf
>>>>>>>>>>>>> /var/ipfire/qos/bin
>>>>>>>>>>>>> /var/ipfire/qos/bin/qos.sh
>>>>>>>>>>>>> /var/ipfire/qos/classes
>>>>>>>>>>>>> /var/ipfire/qos/level7config
>>>>>>>>>>>>> /var/ipfire/qos/portconfig
>>>>>>>>>>>>> /var/ipfire/qos/settings
>>>>>>>>>>>>> /var/ipfire/qos/subclasses
>>>>>>>>>>>>> /var/ipfire/qos/tosconfig
>>>>>>>>>>>>> /var/ipfire/remote/enablessh
>>>>>>>>>>>>> /var/ipfire/remote/settings
>>>>>>>>>>>>> /var/ipfire/sensors/settings
>>>>>>>>>>>>> /var/ipfire/suricata/oinkmaster.conf
>>>>>>>>>>>>> /var/ipfire/suricata/oinkmaster-modify-sids.conf
>>>>>>>>>>>>> /var/ipfire/suricata/oinkmaster-provider-includes.conf
>>>>>>>>>>>>> /var/ipfire/suricata/providers-settings
>>>>>>>>>>>>> /var/ipfire/suricata/settings
>>>>>>>>>>>>> /var/ipfire/suricata/suricata-default-rules.yaml
>>>>>>>>>>>>> /var/ipfire/suricata/suricata-dns-servers.yaml
>>>>>>>>>>>>> /var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
>>>>>>>>>>>>> /var/ipfire/suricata/suricata-homenet.yaml
>>>>>>>>>>>>> /var/ipfire/suricata/suricata-http-ports.yaml
>>>>>>>>>>>>> /var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml
>>>>>>>>>>>>> /var/ipfire/suricata/suricata-used-providers.yaml
>>>>>>>>>>>>> /var/ipfire/time/
>>>>>>>>>>>>> /var/ipfire/time/counter.conf
>>>>>>>>>>>>> /var/ipfire/time/enable
>>>>>>>>>>>>> /var/ipfire/time/settime.conf
>>>>>>>>>>>>> /var/ipfire/time/settings
>>>>>>>>>>>>> /var/ipfire/upnp/settings
>>>>>>>>>>>>> /var/ipfire/urlfilter
>>>>>>>>>>>>> /var/ipfire/urlfilter/settings
>>>>>>>>>>>>> /var/ipfire/urlfilter/squidGuard.conf
>>>>>>>>>>>>> /var/ipfire/vpn
>>>>>>>>>>>>> /var/ipfire/vpn/config
>>>>>>>>>>>>> /var/ipfire/vpn/ipsec.conf
>>>>>>>>>>>>> /var/ipfire/vpn/settings
>>>>>>>>>>>>> /var/ipfire/wakeonlan/clients.conf
>>>>>>>>>>>>> /var/ipfire/wio/wio.conf
>>>>>>>>>>>>> /var/ipfire/wireless/config
>>>>>>>>>>>>> /var/ipfire/wireless/settings
>>>>>>>>>>>>> /var/lib/suricata
>>>>>>>>>>>>> /var/log/rrd/collectd
>>>>>>>>>>>>> /var/log/rrd/hddshutdown-md127.rrd
>>>>>>>>>>>>> /var/log/rrd/hddshutdown-sda.rrd
>>>>>>>>>>>>> /var/log/rrd/hddshutdown-sdb.rrd
>>>>>>>>>>>>> /var/log/rrd/hddtemp-md127.rrd
>>>>>>>>>>>>> /var/log/rrd/hddtemp-sda.rrd
>>>>>>>>>>>>> /var/log/rrd/hddtemp-sdb.rrd
>>>>>>>>>>>>> /var/log/rrd/wio
>>>>>>>>>>>>> /var/log/vnstat
>>>>>>>>>>>>> /var/tmp/idsrules-emerging.tar.gz
>>>>>>>>>>>>> /var/tmp/idsrules-sslbl_blacklist.rules
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Bernhard
>>>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 29 Mar 2022, at 14:10, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Just tried this patch out on my vm testbed system and it still doesn't work for me. The backup file had got to 1.3GB when I deleted the backup file as it was still growing. The normal correct backup file on that vm machine is around 7MB
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The above was the case for both running it from the WUI or from the command line from my unprivileged user using sudo backupctrl exclude
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 29/03/2022 14:27, Michael Tremer wrote:
>>>>>>>>>>>>>>>>> This patch fixes globbing expansion in the backup include file list
>>>>>>>>>>>>>>>>> which got broken in c7e0d73e7cfd7be95db9d0a5f3392b8241813d5b.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>> config/backup/backup.pl | 8 ++++----
>>>>>>>>>>>>>>>>> 1 file changed, 4 insertions(+), 4 deletions(-)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> diff --git a/config/backup/backup.pl b/config/backup/backup.pl
>>>>>>>>>>>>>>>>> index a2337cf23..6f9295e94 100644
>>>>>>>>>>>>>>>>> --- a/config/backup/backup.pl
>>>>>>>>>>>>>>>>> +++ b/config/backup/backup.pl
>>>>>>>>>>>>>>>>> @@ -19,6 +19,8 @@
>>>>>>>>>>>>>>>>> # #
>>>>>>>>>>>>>>>>> ###############################################################################
>>>>>>>>>>>>>>>>> +shopt -s nullglob
>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>> NOW="$(date "+%Y-%m-%d-%H:%M")"
>>>>>>>>>>>>>>>>> list_addons() {
>>>>>>>>>>>>>>>>> @@ -38,10 +40,8 @@ process_includes() {
>>>>>>>>>>>>>>>>> for include in $@; do
>>>>>>>>>>>>>>>>> local file
>>>>>>>>>>>>>>>>> while read -r file; do
>>>>>>>>>>>>>>>>> -  for file in ${file}; do
>>>>>>>>>>>>>>>>> -  if [ -e "/${file}" ]; then
>>>>>>>>>>>>>>>>> -  echo "${file}"
>>>>>>>>>>>>>>>>> -  fi
>>>>>>>>>>>>>>>>> +  for file in /${file}; do
>>>>>>>>>>>>>>>>> +  echo "${file}"
>>>>>>>>>>>>>>>>> done
>>>>>>>>>>>>>>>>> done < "${include}"
>>>>>>>>>>>>>>>>> done | sort -u
>

Michael Tremer March 31, 2022, 10:08 a.m. UTC | #22

Hello,

> On 31 Mar 2022, at 11:02, Bernhard Bitsch <bbitsch@ipfire.org> wrote:
> 
> Hi all,
> 
> yesterday I did some debugging and verification of the function:
> - if you only use the list functionality of backup.pl, you can insert echo commands to see the processing
> - including the root directory for blank lines can be explained looking at this debug output. A blank line gives a file name of length 0, prepending '/' results in '/', which exists.
> - in testing I had a little problem ( which we should keep in mind in analysing problems from the community ); for a quick work flow, I just put the test cases into include.user ( edited on my Windows computer ). But I forgot to check the line ending. So empty lines gave file names of length 1 ( the CR ).
> 
> 
> Some theoretical thoughts.
> Having a background in theoretic informatics and compiler construction the recursive solution with pushd/popd was the most quick and elegant solution for me. On the other hand it is known, that many recursive solutions have an iterative equivalent. For our problem this is Michaels solution. My checks show that both solutions are equivalent, with the check for empty strings.

Yes, they are equivalent and yours worked. I used that for a workaround in the announcement that I just sent out:

  https://blog.ipfire.org/post/incomplete-backups-since-core-update-164

The problem only that it is very easy to forget calling popd (or something it gets skipped because there is a return statement in a function somewhere), and if multiple functions are using this mechanism, this could horribly wrong.

There is no strict requirement to change the current working directory and therefore I felt that we should not use pushd/popd.

But it helped us to narrow down where the problem was. So this was very valuable work.

The final build is now compiling and I am hoping to release it this evening or early tomorrow.

Best,
-Michael

> 
> 
> Regards,
> Bernhard
> 
> Am 31.03.2022 um 09:46 schrieb Michael Tremer:
>> Hello Adolf,
>> Brilliant. Thank you very much for testing this in detail.
>> Best,
>> -Michael
>>> On 31 Mar 2022, at 08:41, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>> 
>>> Hi Michael,
>>> 
>>> On 30/03/2022 23:20, Adolf Belka wrote:
>>>> Hi,
>>>> 
>>>> On 30/03/2022 17:05, Michael Tremer wrote:
>>>>> Hello Bernhard,
>>>>> 
>>>>> You are obviously right. My brain is absolute mush these days and doesn’t allow me to function.
>>>>> 
>>>>> I will correct this by the end of the day.
>>>>> 
>>>>> -Michael
>>>>> 
>>>>>> On 30 Mar 2022, at 15:38, Bernhard Bitsch <bbitsch@ipfire.org> wrote:
>>>>>> 
>>>>>> Hello all,
>>>>>> 
>>>>>> 
>>>>>> Am 30.03.2022 um 16:06 schrieb Michael Tremer:
>>>>>>> Hello everyone,
>>>>>>> Thank you very much for helping me test this.
>>>>>>> I decided that we will have to create an emergency core update for this, because it will otherwise break people’s system. We also have a vulnerability in zlib which is being handed around by the press, so that has helped me to make a decision.
>>>>>> 
>>>>>> I think this is a good solution. Shall we add a warning to backup wiki page? Because in the moment a backup -> reinstall -> restore process leaves an incomplete system.
>>>>>> 
>>>>>>> I also pushed two more patches. One filters out any empty lines because they have quite a damaging effect and it happens easily that they are added to the include list. Secondly, I brought back the existence check which I thought was only there for the globbing mechanism, but also had the small side-effect of filtering out the file list early on which helps keeping tar happy.
>>>>>>> Please review those changes and please install the new c166 from testing as soon as the build has finished (I am not sure if you are all on the nightly-builds list to see any notifications).
>>>> I am not on the notifications list but I will check first thing tomorrow and when the latest has been updated I will do a test install and evaluation of the backup and provide confirmation of how it functions.
>>>> 
>>> As CU166 has been moved to master I took a clone of my CU165 vm and then did a CU on Testing with it rather than doing a fresh install. If you would also want a test of a fresh install done I can also do that.
>>> 
>>> The version I tested was IPFire 2.27 (x86_64) - Core Update 166 Development Build: master/8f696f60
>>> 
>>> Anyway the backup is working fine again. All the directories are present. The size of the backup is now very similar to previously also confirming all contents should be good.
>>> 
>>> I also added back in a couple of blank lines into my include.user file and the backup worked without any problems.
>>> 
>>> I also did a restore of the backup and again as far as I can tell with a quick check through on my testbed, everything is fine.
>>> 
>>> So everything looks good to me.
>>> 
>>> Regards,
>>> 
>>> Adolf.
>>> 
>>>> Regards,
>>>> Adolf.
>>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=4f0e7f24f293ca10096e616010cfd4417afc157c
>>>>>> The test on empty string should be in the 'while read -r file' loop. The empty lines are in the include files. The outer for loop iterates over the file list.
>>>>>> 
>>>>>> Regards,
>>>>>> Bernhard
>>>>>> 
>>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=b275771fdd10ac7116e5d400dc10d3148df8ac27
>>>>>>> Best,
>>>>>>> -Michael
>>>>>>>> On 29 Mar 2022, at 20:21, Bernhard Bitsch <bbitsch@ipfire.org> wrote:
>>>>>>>> 
>>>>>>>> Just checked the blank line problem.
>>>>>>>> Pattern is empty , file name list is '/' --> no names produced ( with existence check )
>>>>>>>> Pattern is empty --> produced file name '/' ( without -e check )
>>>>>>>> 
>>>>>>>> Am 29.03.2022 um 20:36 schrieb Bernhard Bitsch:
>>>>>>>>> Hi all,
>>>>>>>>> Am 29.03.2022 um 17:23 schrieb Adolf Belka:
>>>>>>>>>> Hi All,
>>>>>>>>>> 
>>>>>>>>>> On 29/03/2022 16:22, Michael Tremer wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> On 29 Mar 2022, at 15:22, Bernhard Bitsch <bbitsch@ipfire.org> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> Looks like your include.user file contains absolute paths.
>>>>>>>>>> That was not the problem but I have corrected that. I would have expected that when the main include and exclude files were made relative that the update script would also have checked for any entries in the include.user and exclude.user files but it looks like it didn't.th
>>>>>>>>> According to the source they are chosen.
>>>>>>>>>>> 
>>>>>>>>>>> Or an empty line maybe?
>>>>>>>>>> That turned out to be the problem. There was an empty line after the two lines I had added. You only saw it if you scrolled the pointer down the file in the editor.
>>>>>>>>> Interesting. A blank line should produce nothing.
>>>>>>>>>> 
>>>>>>>>>> Removing that blank line removed the line only with a / and the backup then successfully ran and created a backup file containing all the directories expected.
>>>>>>>>>> 
>>>>>>>>>> The only thing I found is that there was the following tar error message
>>>>>>>>>> 
>>>>>>>>>> tar: Exiting with failure status due to previous errors
>>>>>>>>>> 
>>>>>>>>>> The only thing I found in the backup output was several lines such as
>>>>>>>>>> 
>>>>>>>>>> tar: /etc/squid/squid.conf.local: Cannot stat: No such file or directory
>>>>>>>>>> tar: /etc/squid/squid.conf.pre.local: Cannot stat: No such file or directory
>>>>>>>>>> 
>>>>>>>>>> There were seven lines in total like this and the three I checked were files that did not exist on my system. I presume that these are files that should be backed up if present but don't have to be present so that the error messages are nothing to worry about.
>>>>>>>>>> There were no error messages when doing the backup via the WUI.
>>>>>>>>>> 
>>>>>>>>> Good find. ;)
>>>>>>>>> Michael's patch removes the test of existence, globbing produces existent file names only. But I suppose there are several full names of files, not existing in each implementation.
>>>>>>>>> Conclusion: Repair the file globbing by the shopt and do the existence check. So all existent files are chosen, that are listed in the include files.
>>>>>>>>> Regards,
>>>>>>>>> Bernhard
>>>>>>>>>> The blank line was only on my vm testbed system but as I clone this for any testing activities I will have to check all of my existing vm's to fix this.
>>>>>>>>>> 
>>>>>>>>>> Glad we resolved this simply. Sorry for the anxiety.
>>>>>>>>>> 
>>>>>>>>>> Regards,
>>>>>>>>>> 
>>>>>>>>>> Adolf.
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> Am 29.03.2022 um 16:14 schrieb Adolf Belka:
>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>> This is what I get with the pushd/popd version in place on the same system
>>>>>>>>>>>>> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
>>>>>>>>>>>>> etc/group
>>>>>>>>>>>>> etc/hosts
>>>>>>>>>>>>> etc/hosts.allow
>>>>>>>>>>>>> etc/hosts.deny
>>>>>>>>>>>>> etc/httpd/server.crt
>>>>>>>>>>>>> etc/httpd/server.csr
>>>>>>>>>>>>> etc/httpd/server-ecdsa.crt
>>>>>>>>>>>>> etc/httpd/server-ecdsa.csr
>>>>>>>>>>>>> etc/httpd/server-ecdsa.key
>>>>>>>>>>>>> etc/httpd/server.key
>>>>>>>>>>>>> etc/ipsec.user.conf
>>>>>>>>>>>>> etc/ipsec.user.secrets
>>>>>>>>>>>>> etc/logrotate.d
>>>>>>>>>>>>> etc/passwd
>>>>>>>>>>>>> etc/shadow
>>>>>>>>>>>>> etc/ssh/sshd_config
>>>>>>>>>>>>> etc/ssh/ssh_host_ecdsa_key
>>>>>>>>>>>>> etc/ssh/ssh_host_ecdsa_key.pub
>>>>>>>>>>>>> etc/ssh/ssh_host_ed25519_key
>>>>>>>>>>>>> etc/ssh/ssh_host_ed25519_key.pub
>>>>>>>>>>>>> etc/ssh/ssh_host_rsa_key
>>>>>>>>>>>>> etc/ssh/ssh_host_rsa_key.pub
>>>>>>>>>>>>> /etc/sudoers
>>>>>>>>>>>>> etc/sysconfig/createfiles
>>>>>>>>>>>>> etc/sysconfig/firewall.local
>>>>>>>>>>>>> etc/sysconfig/lm_sensors
>>>>>>>>>>>>> etc/sysconfig/modules
>>>>>>>>>>>>> etc/sysconfig/ramdisk
>>>>>>>>>>>>> etc/sysconfig/rc
>>>>>>>>>>>>> etc/sysconfig/rc.local
>>>>>>>>>>>>> etc/unbound
>>>>>>>>>>>>> /home/ahb
>>>>>>>>>>>>> root/.bash_history
>>>>>>>>>>>>> var/ipfire/accounting/settings.conf
>>>>>>>>>>>>> var/ipfire/auth/users
>>>>>>>>>>>>> var/ipfire/backup/addons/backup
>>>>>>>>>>>>> var/ipfire/backup/exclude.user
>>>>>>>>>>>>> var/ipfire/backup/include.user
>>>>>>>>>>>>> var/ipfire/ca/cacert.pem
>>>>>>>>>>>>> var/ipfire/captive/agb.txt
>>>>>>>>>>>>> var/ipfire/captive/clients
>>>>>>>>>>>>> var/ipfire/captive/coupons
>>>>>>>>>>>>> var/ipfire/captive/logo.dat
>>>>>>>>>>>>> var/ipfire/captive/settings
>>>>>>>>>>>>> var/ipfire/captive/terms.txt
>>>>>>>>>>>>> var/ipfire/captive/voucher_out
>>>>>>>>>>>>> var/ipfire/certs/hostcert.pem
>>>>>>>>>>>>> var/ipfire/certs/hostkey.pem
>>>>>>>>>>>>> var/ipfire/certs/phoebevmipseccert.pem
>>>>>>>>>>>>> var/ipfire/connscheduler/connscheduler.conf
>>>>>>>>>>>>> var/ipfire/crls/cacrl.pem
>>>>>>>>>>>>> var/ipfire/cups/cups-browsed.conf
>>>>>>>>>>>>> var/ipfire/cups/subscriptions.conf
>>>>>>>>>>>>> var/ipfire/ddns/config
>>>>>>>>>>>>> var/ipfire/ddns/ddns.conf
>>>>>>>>>>>>> var/ipfire/ddns/settings
>>>>>>>>>>>>> var/ipfire/dhcp/advoptions
>>>>>>>>>>>>> var/ipfire/dhcp/advoptions-list
>>>>>>>>>>>>> var/ipfire/dhcpc/dhcpcd.conf
>>>>>>>>>>>>> var/ipfire/dhcp/dhcpd.conf
>>>>>>>>>>>>> var/ipfire/dhcp/dhcpd.conf.local
>>>>>>>>>>>>> var/ipfire/dhcp/enable_blue
>>>>>>>>>>>>> var/ipfire/dhcp/enable_green
>>>>>>>>>>>>> var/ipfire/dhcp/fixleases
>>>>>>>>>>>>> var/ipfire/dhcp/settings
>>>>>>>>>>>>> var/ipfire/dma/auth.conf
>>>>>>>>>>>>> var/ipfire/dma/dma.conf
>>>>>>>>>>>>> var/ipfire/dma/mail.conf
>>>>>>>>>>>>> var/ipfire/dns
>>>>>>>>>>>>> var/ipfire/dnsforward/config
>>>>>>>>>>>>> var/ipfire/dns/settings
>>>>>>>>>>>>> var/ipfire/ethernet/aliases
>>>>>>>>>>>>> var/ipfire/ethernet/settings
>>>>>>>>>>>>> var/ipfire/ethernet/wireless
>>>>>>>>>>>>> var/ipfire/extrahd/settings
>>>>>>>>>>>>> var/ipfire/firewall
>>>>>>>>>>>>> var/ipfire/firewall/config
>>>>>>>>>>>>> var/ipfire/firewall/settings
>>>>>>>>>>>>> var/ipfire/fwhosts
>>>>>>>>>>>>> var/ipfire/isdn/settings
>>>>>>>>>>>>> var/ipfire/logging/settings
>>>>>>>>>>>>> var/ipfire/mac/settings
>>>>>>>>>>>>> var/ipfire/main/firstsetup_ok
>>>>>>>>>>>>> var/ipfire/main/gpl_accepted
>>>>>>>>>>>>> var/ipfire/main/hostname.conf
>>>>>>>>>>>>> var/ipfire/main/hosts
>>>>>>>>>>>>> var/ipfire/main/manualpages
>>>>>>>>>>>>> var/ipfire/main/routing
>>>>>>>>>>>>> var/ipfire/main/security
>>>>>>>>>>>>> var/ipfire/main/send_profile
>>>>>>>>>>>>> var/ipfire/main/settings
>>>>>>>>>>>>> var/ipfire/modem/settings
>>>>>>>>>>>>> var/ipfire/optionsfw/settings
>>>>>>>>>>>>> var/ipfire/ovpn
>>>>>>>>>>>>> var/ipfire/ovpn/ccd.conf
>>>>>>>>>>>>> var/ipfire/ovpn/collectd.vpn
>>>>>>>>>>>>> var/ipfire/ovpn/enable
>>>>>>>>>>>>> var/ipfire/ovpn/server.conf
>>>>>>>>>>>>> var/ipfire/ovpn/settings
>>>>>>>>>>>>> var/ipfire/pakfire/settings
>>>>>>>>>>>>> var/ipfire/ppp
>>>>>>>>>>>>> var/ipfire/ppp/fake-resolv.conf
>>>>>>>>>>>>> var/ipfire/ppp/settings
>>>>>>>>>>>>> var/ipfire/private/cakey.pem
>>>>>>>>>>>>> var/ipfire/proxy
>>>>>>>>>>>>> var/ipfire/proxy/asnbl-helper.conf
>>>>>>>>>>>>> var/ipfire/proxy/cachemgr.conf
>>>>>>>>>>>>> var/ipfire/proxy/enable
>>>>>>>>>>>>> var/ipfire/proxy/settings
>>>>>>>>>>>>> var/ipfire/proxy/squid.conf
>>>>>>>>>>>>> var/ipfire/qos/bin
>>>>>>>>>>>>> var/ipfire/qos/classes
>>>>>>>>>>>>> var/ipfire/qos/level7config
>>>>>>>>>>>>> var/ipfire/qos/portconfig
>>>>>>>>>>>>> var/ipfire/qos/settings
>>>>>>>>>>>>> var/ipfire/qos/subclasses
>>>>>>>>>>>>> var/ipfire/qos/tosconfig
>>>>>>>>>>>>> var/ipfire/remote/enablessh
>>>>>>>>>>>>> var/ipfire/remote/settings
>>>>>>>>>>>>> var/ipfire/sensors/settings
>>>>>>>>>>>>> var/ipfire/suricata/oinkmaster.conf
>>>>>>>>>>>>> var/ipfire/suricata/oinkmaster-modify-sids.conf
>>>>>>>>>>>>> var/ipfire/suricata/oinkmaster-provider-includes.conf
>>>>>>>>>>>>> var/ipfire/suricata/providers-settings
>>>>>>>>>>>>> var/ipfire/suricata/settings
>>>>>>>>>>>>> var/ipfire/suricata/suricata-default-rules.yaml
>>>>>>>>>>>>> var/ipfire/suricata/suricata-dns-servers.yaml
>>>>>>>>>>>>> var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
>>>>>>>>>>>>> var/ipfire/suricata/suricata-homenet.yaml
>>>>>>>>>>>>> var/ipfire/suricata/suricata-http-ports.yaml
>>>>>>>>>>>>> var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml
>>>>>>>>>>>>> var/ipfire/suricata/suricata-used-providers.yaml
>>>>>>>>>>>>> var/ipfire/time/
>>>>>>>>>>>>> var/ipfire/time/counter.conf
>>>>>>>>>>>>> var/ipfire/time/enable
>>>>>>>>>>>>> var/ipfire/time/settime.conf
>>>>>>>>>>>>> var/ipfire/time/settings
>>>>>>>>>>>>> var/ipfire/upnp/settings
>>>>>>>>>>>>> var/ipfire/urlfilter
>>>>>>>>>>>>> var/ipfire/urlfilter/settings
>>>>>>>>>>>>> var/ipfire/urlfilter/squidGuard.conf
>>>>>>>>>>>>> var/ipfire/vpn
>>>>>>>>>>>>> var/ipfire/vpn/config
>>>>>>>>>>>>> var/ipfire/vpn/ipsec.conf
>>>>>>>>>>>>> var/ipfire/vpn/settings
>>>>>>>>>>>>> var/ipfire/wakeonlan/clients.conf
>>>>>>>>>>>>> var/ipfire/wio/wio.conf
>>>>>>>>>>>>> var/ipfire/wireless/config
>>>>>>>>>>>>> var/ipfire/wireless/settings
>>>>>>>>>>>>> var/lib/suricata
>>>>>>>>>>>>> var/log/rrd/collectd
>>>>>>>>>>>>> var/log/rrd/hddshutdown-md127.rrd
>>>>>>>>>>>>> var/log/rrd/hddshutdown-sda.rrd
>>>>>>>>>>>>> var/log/rrd/hddshutdown-sdb.rrd
>>>>>>>>>>>>> var/log/rrd/hddtemp-md127.rrd
>>>>>>>>>>>>> var/log/rrd/hddtemp-sda.rrd
>>>>>>>>>>>>> var/log/rrd/hddtemp-sdb.rrd
>>>>>>>>>>>>> var/log/rrd/wio
>>>>>>>>>>>>> var/log/vnstat
>>>>>>>>>>>>> var/tmp/idsrules-emerging.tar.gz
>>>>>>>>>>>>> var/tmp/idsrules-sslbl_blacklist.rules
>>>>>>>>>>>>> The following are in the previous list but not in this one:-
>>>>>>>>>>>>> /
>>>>>>>>>>>>> /etc/conntrackd/conntrackd.conf
>>>>>>>>>>>>> /etc/ipsec.user-post.conf
>>>>>>>>>>>>> /root/.gitconfig
>>>>>>>>>>>>> /root/.ssh
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>> On 29/03/2022 16:01, Adolf Belka wrote:
>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On 29/03/2022 15:36, Bernhard Bitsch wrote:
>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Am 29.03.2022 um 15:11 schrieb Michael Tremer:
>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Could you please send the file listing to find out what is being included what shouldn’t?
>>>>>>>>>>>>>> Unfortunately, as I stopped the backup continuing once it had reached 1.2GB, the file created was not able to be opened. Probably stopping the backup corrupted it in some way.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Just for easy quick handling, you can use
>>>>>>>>>>>>>>> '/var/ipfire/backup/bin/backup.pl list' to show the files included.
>>>>>>>>>>>>>> I gave this a go and here is the output from the command.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
>>>>>>>>>>>>>> /
>>>>>>>>>>>>>> /etc/conntrackd/conntrackd.conf
>>>>>>>>>>>>>> /etc/group
>>>>>>>>>>>>>> /etc/hosts
>>>>>>>>>>>>>> /etc/hosts.allow
>>>>>>>>>>>>>> /etc/hosts.deny
>>>>>>>>>>>>>> /etc/httpd/server.crt
>>>>>>>>>>>>>> /etc/httpd/server.csr
>>>>>>>>>>>>>> /etc/httpd/server-ecdsa.crt
>>>>>>>>>>>>>> /etc/httpd/server-ecdsa.csr
>>>>>>>>>>>>>> /etc/httpd/server-ecdsa.key
>>>>>>>>>>>>>> /etc/httpd/server.key
>>>>>>>>>>>>>> /etc/ipsec.user.conf
>>>>>>>>>>>>>> /etc/ipsec.user-post.conf
>>>>>>>>>>>>>> /etc/ipsec.user.secrets
>>>>>>>>>>>>>> /etc/logrotate.d
>>>>>>>>>>>>>> /etc/passwd
>>>>>>>>>>>>>> /etc/shadow
>>>>>>>>>>>>>> /etc/squid/squid.conf.local
>>>>>>>>>>>>>> /etc/squid/squid.conf.pre.local
>>>>>>>>>>>>>> /etc/ssh/sshd_config
>>>>>>>>>>>>>> /etc/ssh/ssh_host_ecdsa_key
>>>>>>>>>>>>>> /etc/ssh/ssh_host_ecdsa_key.pub
>>>>>>>>>>>>>> /etc/ssh/ssh_host_ed25519_key
>>>>>>>>>>>>>> /etc/ssh/ssh_host_ed25519_key.pub
>>>>>>>>>>>>>> /etc/ssh/ssh_host_rsa_key
>>>>>>>>>>>>>> /etc/ssh/ssh_host_rsa_key.pub
>>>>>>>>>>>>>> //etc/sudoers
>>>>>>>>>>>>>> /etc/sysconfig/createfiles
>>>>>>>>>>>>>> /etc/sysconfig/firewall.local
>>>>>>>>>>>>>> /etc/sysconfig/lm_sensors
>>>>>>>>>>>>>> /etc/sysconfig/modules
>>>>>>>>>>>>>> /etc/sysconfig/ramdisk
>>>>>>>>>>>>>> /etc/sysconfig/rc
>>>>>>>>>>>>>> /etc/sysconfig/rc.local
>>>>>>>>>>>>>> /etc/unbound
>>>>>>>>>>>>>> //home/ahb
>>>>>>>>>>>>>> /root/.bash_history
>>>>>>>>>>>>>> /root/.gitconfig
>>>>>>>>>>>>>> /root/.ssh
>>>>>>>>>>>>>> /var/ipfire/accounting/settings.conf
>>>>>>>>>>>>>> /var/ipfire/auth/users
>>>>>>>>>>>>>> /var/ipfire/backup/addons/backup
>>>>>>>>>>>>>> /var/ipfire/backup/exclude.user
>>>>>>>>>>>>>> /var/ipfire/backup/include.user
>>>>>>>>>>>>>> /var/ipfire/ca/cacert.pem
>>>>>>>>>>>>>> /var/ipfire/captive/agb.txt
>>>>>>>>>>>>>> /var/ipfire/captive/clients
>>>>>>>>>>>>>> /var/ipfire/captive/coupons
>>>>>>>>>>>>>> /var/ipfire/captive/logo.dat
>>>>>>>>>>>>>> /var/ipfire/captive/settings
>>>>>>>>>>>>>> /var/ipfire/captive/terms.txt
>>>>>>>>>>>>>> /var/ipfire/captive/voucher_out
>>>>>>>>>>>>>> /var/ipfire/certs/hostcert.pem
>>>>>>>>>>>>>> /var/ipfire/certs/hostkey.pem
>>>>>>>>>>>>>> /var/ipfire/certs/phoebevmipseccert.pem
>>>>>>>>>>>>>> /var/ipfire/connscheduler/connscheduler.conf
>>>>>>>>>>>>>> /var/ipfire/crls/cacrl.pem
>>>>>>>>>>>>>> /var/ipfire/cups/cups-browsed.conf
>>>>>>>>>>>>>> /var/ipfire/cups/subscriptions.conf
>>>>>>>>>>>>>> /var/ipfire/ddns/config
>>>>>>>>>>>>>> /var/ipfire/ddns/ddns.conf
>>>>>>>>>>>>>> /var/ipfire/ddns/settings
>>>>>>>>>>>>>> /var/ipfire/dhcp/advoptions
>>>>>>>>>>>>>> /var/ipfire/dhcp/advoptions-list
>>>>>>>>>>>>>> /var/ipfire/dhcpc/dhcpcd.conf
>>>>>>>>>>>>>> /var/ipfire/dhcp/dhcpd.conf
>>>>>>>>>>>>>> /var/ipfire/dhcp/dhcpd.conf.local
>>>>>>>>>>>>>> /var/ipfire/dhcp/enable_blue
>>>>>>>>>>>>>> /var/ipfire/dhcp/enable_green
>>>>>>>>>>>>>> /var/ipfire/dhcp/fixleases
>>>>>>>>>>>>>> /var/ipfire/dhcp/settings
>>>>>>>>>>>>>> /var/ipfire/dma/auth.conf
>>>>>>>>>>>>>> /var/ipfire/dma/dma.conf
>>>>>>>>>>>>>> /var/ipfire/dma/mail.conf
>>>>>>>>>>>>>> /var/ipfire/dns
>>>>>>>>>>>>>> /var/ipfire/dnsforward/config
>>>>>>>>>>>>>> /var/ipfire/dns/settings
>>>>>>>>>>>>>> /var/ipfire/ethernet/aliases
>>>>>>>>>>>>>> /var/ipfire/ethernet/settings
>>>>>>>>>>>>>> /var/ipfire/ethernet/wireless
>>>>>>>>>>>>>> /var/ipfire/extrahd/settings
>>>>>>>>>>>>>> /var/ipfire/firewall
>>>>>>>>>>>>>> /var/ipfire/firewall/config
>>>>>>>>>>>>>> /var/ipfire/firewall/settings
>>>>>>>>>>>>>> /var/ipfire/fwhosts
>>>>>>>>>>>>>> /var/ipfire/isdn/settings
>>>>>>>>>>>>>> /var/ipfire/logging/settings
>>>>>>>>>>>>>> /var/ipfire/mac/settings
>>>>>>>>>>>>>> /var/ipfire/main/firstsetup_ok
>>>>>>>>>>>>>> /var/ipfire/main/gpl_accepted
>>>>>>>>>>>>>> /var/ipfire/main/hostname.conf
>>>>>>>>>>>>>> /var/ipfire/main/hosts
>>>>>>>>>>>>>> /var/ipfire/main/manualpages
>>>>>>>>>>>>>> /var/ipfire/main/routing
>>>>>>>>>>>>>> /var/ipfire/main/security
>>>>>>>>>>>>>> /var/ipfire/main/send_profile
>>>>>>>>>>>>>> /var/ipfire/main/settings
>>>>>>>>>>>>>> /var/ipfire/modem/settings
>>>>>>>>>>>>>> /var/ipfire/optionsfw/settings
>>>>>>>>>>>>>> /var/ipfire/ovpn
>>>>>>>>>>>>>> /var/ipfire/ovpn/ccd.conf
>>>>>>>>>>>>>> /var/ipfire/ovpn/collectd.vpn
>>>>>>>>>>>>>> /var/ipfire/ovpn/enable
>>>>>>>>>>>>>> /var/ipfire/ovpn/server.conf
>>>>>>>>>>>>>> /var/ipfire/ovpn/settings
>>>>>>>>>>>>>> /var/ipfire/pakfire/settings
>>>>>>>>>>>>>> /var/ipfire/ppp
>>>>>>>>>>>>>> /var/ipfire/ppp/fake-resolv.conf
>>>>>>>>>>>>>> /var/ipfire/ppp/settings
>>>>>>>>>>>>>> /var/ipfire/private/cakey.pem
>>>>>>>>>>>>>> /var/ipfire/proxy
>>>>>>>>>>>>>> /var/ipfire/proxy/asnbl-helper.conf
>>>>>>>>>>>>>> /var/ipfire/proxy/cachemgr.conf
>>>>>>>>>>>>>> /var/ipfire/proxy/enable
>>>>>>>>>>>>>> /var/ipfire/proxy/settings
>>>>>>>>>>>>>> /var/ipfire/proxy/squid.conf
>>>>>>>>>>>>>> /var/ipfire/qos/bin
>>>>>>>>>>>>>> /var/ipfire/qos/bin/qos.sh
>>>>>>>>>>>>>> /var/ipfire/qos/classes
>>>>>>>>>>>>>> /var/ipfire/qos/level7config
>>>>>>>>>>>>>> /var/ipfire/qos/portconfig
>>>>>>>>>>>>>> /var/ipfire/qos/settings
>>>>>>>>>>>>>> /var/ipfire/qos/subclasses
>>>>>>>>>>>>>> /var/ipfire/qos/tosconfig
>>>>>>>>>>>>>> /var/ipfire/remote/enablessh
>>>>>>>>>>>>>> /var/ipfire/remote/settings
>>>>>>>>>>>>>> /var/ipfire/sensors/settings
>>>>>>>>>>>>>> /var/ipfire/suricata/oinkmaster.conf
>>>>>>>>>>>>>> /var/ipfire/suricata/oinkmaster-modify-sids.conf
>>>>>>>>>>>>>> /var/ipfire/suricata/oinkmaster-provider-includes.conf
>>>>>>>>>>>>>> /var/ipfire/suricata/providers-settings
>>>>>>>>>>>>>> /var/ipfire/suricata/settings
>>>>>>>>>>>>>> /var/ipfire/suricata/suricata-default-rules.yaml
>>>>>>>>>>>>>> /var/ipfire/suricata/suricata-dns-servers.yaml
>>>>>>>>>>>>>> /var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
>>>>>>>>>>>>>> /var/ipfire/suricata/suricata-homenet.yaml
>>>>>>>>>>>>>> /var/ipfire/suricata/suricata-http-ports.yaml
>>>>>>>>>>>>>> /var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml
>>>>>>>>>>>>>> /var/ipfire/suricata/suricata-used-providers.yaml
>>>>>>>>>>>>>> /var/ipfire/time/
>>>>>>>>>>>>>> /var/ipfire/time/counter.conf
>>>>>>>>>>>>>> /var/ipfire/time/enable
>>>>>>>>>>>>>> /var/ipfire/time/settime.conf
>>>>>>>>>>>>>> /var/ipfire/time/settings
>>>>>>>>>>>>>> /var/ipfire/upnp/settings
>>>>>>>>>>>>>> /var/ipfire/urlfilter
>>>>>>>>>>>>>> /var/ipfire/urlfilter/settings
>>>>>>>>>>>>>> /var/ipfire/urlfilter/squidGuard.conf
>>>>>>>>>>>>>> /var/ipfire/vpn
>>>>>>>>>>>>>> /var/ipfire/vpn/config
>>>>>>>>>>>>>> /var/ipfire/vpn/ipsec.conf
>>>>>>>>>>>>>> /var/ipfire/vpn/settings
>>>>>>>>>>>>>> /var/ipfire/wakeonlan/clients.conf
>>>>>>>>>>>>>> /var/ipfire/wio/wio.conf
>>>>>>>>>>>>>> /var/ipfire/wireless/config
>>>>>>>>>>>>>> /var/ipfire/wireless/settings
>>>>>>>>>>>>>> /var/lib/suricata
>>>>>>>>>>>>>> /var/log/rrd/collectd
>>>>>>>>>>>>>> /var/log/rrd/hddshutdown-md127.rrd
>>>>>>>>>>>>>> /var/log/rrd/hddshutdown-sda.rrd
>>>>>>>>>>>>>> /var/log/rrd/hddshutdown-sdb.rrd
>>>>>>>>>>>>>> /var/log/rrd/hddtemp-md127.rrd
>>>>>>>>>>>>>> /var/log/rrd/hddtemp-sda.rrd
>>>>>>>>>>>>>> /var/log/rrd/hddtemp-sdb.rrd
>>>>>>>>>>>>>> /var/log/rrd/wio
>>>>>>>>>>>>>> /var/log/vnstat
>>>>>>>>>>>>>> /var/tmp/idsrules-emerging.tar.gz
>>>>>>>>>>>>>> /var/tmp/idsrules-sslbl_blacklist.rules
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Bernhard
>>>>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On 29 Mar 2022, at 14:10, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Just tried this patch out on my vm testbed system and it still doesn't work for me. The backup file had got to 1.3GB when I deleted the backup file as it was still growing. The normal correct backup file on that vm machine is around 7MB
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> The above was the case for both running it from the WUI or from the command line from my unprivileged user using sudo backupctrl exclude
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On 29/03/2022 14:27, Michael Tremer wrote:
>>>>>>>>>>>>>>>>>> This patch fixes globbing expansion in the backup include file list
>>>>>>>>>>>>>>>>>> which got broken in c7e0d73e7cfd7be95db9d0a5f3392b8241813d5b.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>> config/backup/backup.pl | 8 ++++----
>>>>>>>>>>>>>>>>>> 1 file changed, 4 insertions(+), 4 deletions(-)
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> diff --git a/config/backup/backup.pl b/config/backup/backup.pl
>>>>>>>>>>>>>>>>>> index a2337cf23..6f9295e94 100644
>>>>>>>>>>>>>>>>>> --- a/config/backup/backup.pl
>>>>>>>>>>>>>>>>>> +++ b/config/backup/backup.pl
>>>>>>>>>>>>>>>>>> @@ -19,6 +19,8 @@
>>>>>>>>>>>>>>>>>> # #
>>>>>>>>>>>>>>>>>> ###############################################################################
>>>>>>>>>>>>>>>>>> +shopt -s nullglob
>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>> NOW="$(date "+%Y-%m-%d-%H:%M")"
>>>>>>>>>>>>>>>>>> list_addons() {
>>>>>>>>>>>>>>>>>> @@ -38,10 +40,8 @@ process_includes() {
>>>>>>>>>>>>>>>>>> for include in $@; do
>>>>>>>>>>>>>>>>>> local file
>>>>>>>>>>>>>>>>>> while read -r file; do
>>>>>>>>>>>>>>>>>> -  for file in ${file}; do
>>>>>>>>>>>>>>>>>> -  if [ -e "/${file}" ]; then
>>>>>>>>>>>>>>>>>> -  echo "${file}"
>>>>>>>>>>>>>>>>>> -  fi
>>>>>>>>>>>>>>>>>> +  for file in /${file}; do
>>>>>>>>>>>>>>>>>> +  echo "${file}"
>>>>>>>>>>>>>>>>>> done
>>>>>>>>>>>>>>>>>> done < "${include}"
>>>>>>>>>>>>>>>>>> done | sort -u

Adolf Belka March 31, 2022, 10:16 a.m. UTC | #23

Hi Michael,

Update on the testing. I realised that I had done the test on a CU175 version where I had modified the backup.pl code with pushd/popd.
I wouldn't expect that to give a problem but to confirm I created another CU165 clone with the backup.pl code in original form as most standard users would have it.
Re-ran the test and everything went correctly the same as my previous test.

Just for security I also did a fresh install of CU166 from the latest master nightly.
That also worked fine. The backup had all the directories as expected.

Regards,
Adolf

On 31/03/2022 09:46, Michael Tremer wrote:
> Hello Adolf,
> 
> Brilliant. Thank you very much for testing this in detail.
> 
> Best,
> -Michael
> 
>> On 31 Mar 2022, at 08:41, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>
>> Hi Michael,
>>
>> On 30/03/2022 23:20, Adolf Belka wrote:
>>> Hi,
>>>
>>> On 30/03/2022 17:05, Michael Tremer wrote:
>>>> Hello Bernhard,
>>>>
>>>> You are obviously right. My brain is absolute mush these days and doesn’t allow me to function.
>>>>
>>>> I will correct this by the end of the day.
>>>>
>>>> -Michael
>>>>
>>>>> On 30 Mar 2022, at 15:38, Bernhard Bitsch <bbitsch@ipfire.org> wrote:
>>>>>
>>>>> Hello all,
>>>>>
>>>>>
>>>>> Am 30.03.2022 um 16:06 schrieb Michael Tremer:
>>>>>> Hello everyone,
>>>>>> Thank you very much for helping me test this.
>>>>>> I decided that we will have to create an emergency core update for this, because it will otherwise break people’s system. We also have a vulnerability in zlib which is being handed around by the press, so that has helped me to make a decision.
>>>>>
>>>>> I think this is a good solution. Shall we add a warning to backup wiki page? Because in the moment a backup -> reinstall -> restore process leaves an incomplete system.
>>>>>
>>>>>> I also pushed two more patches. One filters out any empty lines because they have quite a damaging effect and it happens easily that they are added to the include list. Secondly, I brought back the existence check which I thought was only there for the globbing mechanism, but also had the small side-effect of filtering out the file list early on which helps keeping tar happy.
>>>>>> Please review those changes and please install the new c166 from testing as soon as the build has finished (I am not sure if you are all on the nightly-builds list to see any notifications).
>>> I am not on the notifications list but I will check first thing tomorrow and when the latest has been updated I will do a test install and evaluation of the backup and provide confirmation of how it functions.
>>>
>> As CU166 has been moved to master I took a clone of my CU165 vm and then did a CU on Testing with it rather than doing a fresh install. If you would also want a test of a fresh install done I can also do that.
>>
>> The version I tested was IPFire 2.27 (x86_64) - Core Update 166 Development Build: master/8f696f60
>>
>> Anyway the backup is working fine again. All the directories are present. The size of the backup is now very similar to previously also confirming all contents should be good.
>>
>> I also added back in a couple of blank lines into my include.user file and the backup worked without any problems.
>>
>> I also did a restore of the backup and again as far as I can tell with a quick check through on my testbed, everything is fine.
>>
>> So everything looks good to me.
>>
>> Regards,
>>
>> Adolf.
>>
>>> Regards,
>>> Adolf.
>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=4f0e7f24f293ca10096e616010cfd4417afc157c
>>>>> The test on empty string should be in the 'while read -r file' loop. The empty lines are in the include files. The outer for loop iterates over the file list.
>>>>>
>>>>> Regards,
>>>>> Bernhard
>>>>>
>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=b275771fdd10ac7116e5d400dc10d3148df8ac27
>>>>>> Best,
>>>>>> -Michael
>>>>>>> On 29 Mar 2022, at 20:21, Bernhard Bitsch <bbitsch@ipfire.org> wrote:
>>>>>>>
>>>>>>> Just checked the blank line problem.
>>>>>>> Pattern is empty , file name list is '/' --> no names produced ( with existence check )
>>>>>>> Pattern is empty --> produced file name '/' ( without -e check )
>>>>>>>
>>>>>>> Am 29.03.2022 um 20:36 schrieb Bernhard Bitsch:
>>>>>>>> Hi all,
>>>>>>>> Am 29.03.2022 um 17:23 schrieb Adolf Belka:
>>>>>>>>> Hi All,
>>>>>>>>>
>>>>>>>>> On 29/03/2022 16:22, Michael Tremer wrote:
>>>>>>>>>>
>>>>>>>>>>> On 29 Mar 2022, at 15:22, Bernhard Bitsch <bbitsch@ipfire.org> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Looks like your include.user file contains absolute paths.
>>>>>>>>> That was not the problem but I have corrected that. I would have expected that when the main include and exclude files were made relative that the update script would also have checked for any entries in the include.user and exclude.user files but it looks like it didn't.th
>>>>>>>> According to the source they are chosen.
>>>>>>>>>>
>>>>>>>>>> Or an empty line maybe?
>>>>>>>>> That turned out to be the problem. There was an empty line after the two lines I had added. You only saw it if you scrolled the pointer down the file in the editor.
>>>>>>>> Interesting. A blank line should produce nothing.
>>>>>>>>>
>>>>>>>>> Removing that blank line removed the line only with a / and the backup then successfully ran and created a backup file containing all the directories expected.
>>>>>>>>>
>>>>>>>>> The only thing I found is that there was the following tar error message
>>>>>>>>>
>>>>>>>>> tar: Exiting with failure status due to previous errors
>>>>>>>>>
>>>>>>>>> The only thing I found in the backup output was several lines such as
>>>>>>>>>
>>>>>>>>> tar: /etc/squid/squid.conf.local: Cannot stat: No such file or directory
>>>>>>>>> tar: /etc/squid/squid.conf.pre.local: Cannot stat: No such file or directory
>>>>>>>>>
>>>>>>>>> There were seven lines in total like this and the three I checked were files that did not exist on my system. I presume that these are files that should be backed up if present but don't have to be present so that the error messages are nothing to worry about.
>>>>>>>>> There were no error messages when doing the backup via the WUI.
>>>>>>>>>
>>>>>>>> Good find. ;)
>>>>>>>> Michael's patch removes the test of existence, globbing produces existent file names only. But I suppose there are several full names of files, not existing in each implementation.
>>>>>>>> Conclusion: Repair the file globbing by the shopt and do the existence check. So all existent files are chosen, that are listed in the include files.
>>>>>>>> Regards,
>>>>>>>> Bernhard
>>>>>>>>> The blank line was only on my vm testbed system but as I clone this for any testing activities I will have to check all of my existing vm's to fix this.
>>>>>>>>>
>>>>>>>>> Glad we resolved this simply. Sorry for the anxiety.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Adolf.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Am 29.03.2022 um 16:14 schrieb Adolf Belka:
>>>>>>>>>>>> Hi All,
>>>>>>>>>>>> This is what I get with the pushd/popd version in place on the same system
>>>>>>>>>>>> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
>>>>>>>>>>>> etc/group
>>>>>>>>>>>> etc/hosts
>>>>>>>>>>>> etc/hosts.allow
>>>>>>>>>>>> etc/hosts.deny
>>>>>>>>>>>> etc/httpd/server.crt
>>>>>>>>>>>> etc/httpd/server.csr
>>>>>>>>>>>> etc/httpd/server-ecdsa.crt
>>>>>>>>>>>> etc/httpd/server-ecdsa.csr
>>>>>>>>>>>> etc/httpd/server-ecdsa.key
>>>>>>>>>>>> etc/httpd/server.key
>>>>>>>>>>>> etc/ipsec.user.conf
>>>>>>>>>>>> etc/ipsec.user.secrets
>>>>>>>>>>>> etc/logrotate.d
>>>>>>>>>>>> etc/passwd
>>>>>>>>>>>> etc/shadow
>>>>>>>>>>>> etc/ssh/sshd_config
>>>>>>>>>>>> etc/ssh/ssh_host_ecdsa_key
>>>>>>>>>>>> etc/ssh/ssh_host_ecdsa_key.pub
>>>>>>>>>>>> etc/ssh/ssh_host_ed25519_key
>>>>>>>>>>>> etc/ssh/ssh_host_ed25519_key.pub
>>>>>>>>>>>> etc/ssh/ssh_host_rsa_key
>>>>>>>>>>>> etc/ssh/ssh_host_rsa_key.pub
>>>>>>>>>>>> /etc/sudoers
>>>>>>>>>>>> etc/sysconfig/createfiles
>>>>>>>>>>>> etc/sysconfig/firewall.local
>>>>>>>>>>>> etc/sysconfig/lm_sensors
>>>>>>>>>>>> etc/sysconfig/modules
>>>>>>>>>>>> etc/sysconfig/ramdisk
>>>>>>>>>>>> etc/sysconfig/rc
>>>>>>>>>>>> etc/sysconfig/rc.local
>>>>>>>>>>>> etc/unbound
>>>>>>>>>>>> /home/ahb
>>>>>>>>>>>> root/.bash_history
>>>>>>>>>>>> var/ipfire/accounting/settings.conf
>>>>>>>>>>>> var/ipfire/auth/users
>>>>>>>>>>>> var/ipfire/backup/addons/backup
>>>>>>>>>>>> var/ipfire/backup/exclude.user
>>>>>>>>>>>> var/ipfire/backup/include.user
>>>>>>>>>>>> var/ipfire/ca/cacert.pem
>>>>>>>>>>>> var/ipfire/captive/agb.txt
>>>>>>>>>>>> var/ipfire/captive/clients
>>>>>>>>>>>> var/ipfire/captive/coupons
>>>>>>>>>>>> var/ipfire/captive/logo.dat
>>>>>>>>>>>> var/ipfire/captive/settings
>>>>>>>>>>>> var/ipfire/captive/terms.txt
>>>>>>>>>>>> var/ipfire/captive/voucher_out
>>>>>>>>>>>> var/ipfire/certs/hostcert.pem
>>>>>>>>>>>> var/ipfire/certs/hostkey.pem
>>>>>>>>>>>> var/ipfire/certs/phoebevmipseccert.pem
>>>>>>>>>>>> var/ipfire/connscheduler/connscheduler.conf
>>>>>>>>>>>> var/ipfire/crls/cacrl.pem
>>>>>>>>>>>> var/ipfire/cups/cups-browsed.conf
>>>>>>>>>>>> var/ipfire/cups/subscriptions.conf
>>>>>>>>>>>> var/ipfire/ddns/config
>>>>>>>>>>>> var/ipfire/ddns/ddns.conf
>>>>>>>>>>>> var/ipfire/ddns/settings
>>>>>>>>>>>> var/ipfire/dhcp/advoptions
>>>>>>>>>>>> var/ipfire/dhcp/advoptions-list
>>>>>>>>>>>> var/ipfire/dhcpc/dhcpcd.conf
>>>>>>>>>>>> var/ipfire/dhcp/dhcpd.conf
>>>>>>>>>>>> var/ipfire/dhcp/dhcpd.conf.local
>>>>>>>>>>>> var/ipfire/dhcp/enable_blue
>>>>>>>>>>>> var/ipfire/dhcp/enable_green
>>>>>>>>>>>> var/ipfire/dhcp/fixleases
>>>>>>>>>>>> var/ipfire/dhcp/settings
>>>>>>>>>>>> var/ipfire/dma/auth.conf
>>>>>>>>>>>> var/ipfire/dma/dma.conf
>>>>>>>>>>>> var/ipfire/dma/mail.conf
>>>>>>>>>>>> var/ipfire/dns
>>>>>>>>>>>> var/ipfire/dnsforward/config
>>>>>>>>>>>> var/ipfire/dns/settings
>>>>>>>>>>>> var/ipfire/ethernet/aliases
>>>>>>>>>>>> var/ipfire/ethernet/settings
>>>>>>>>>>>> var/ipfire/ethernet/wireless
>>>>>>>>>>>> var/ipfire/extrahd/settings
>>>>>>>>>>>> var/ipfire/firewall
>>>>>>>>>>>> var/ipfire/firewall/config
>>>>>>>>>>>> var/ipfire/firewall/settings
>>>>>>>>>>>> var/ipfire/fwhosts
>>>>>>>>>>>> var/ipfire/isdn/settings
>>>>>>>>>>>> var/ipfire/logging/settings
>>>>>>>>>>>> var/ipfire/mac/settings
>>>>>>>>>>>> var/ipfire/main/firstsetup_ok
>>>>>>>>>>>> var/ipfire/main/gpl_accepted
>>>>>>>>>>>> var/ipfire/main/hostname.conf
>>>>>>>>>>>> var/ipfire/main/hosts
>>>>>>>>>>>> var/ipfire/main/manualpages
>>>>>>>>>>>> var/ipfire/main/routing
>>>>>>>>>>>> var/ipfire/main/security
>>>>>>>>>>>> var/ipfire/main/send_profile
>>>>>>>>>>>> var/ipfire/main/settings
>>>>>>>>>>>> var/ipfire/modem/settings
>>>>>>>>>>>> var/ipfire/optionsfw/settings
>>>>>>>>>>>> var/ipfire/ovpn
>>>>>>>>>>>> var/ipfire/ovpn/ccd.conf
>>>>>>>>>>>> var/ipfire/ovpn/collectd.vpn
>>>>>>>>>>>> var/ipfire/ovpn/enable
>>>>>>>>>>>> var/ipfire/ovpn/server.conf
>>>>>>>>>>>> var/ipfire/ovpn/settings
>>>>>>>>>>>> var/ipfire/pakfire/settings
>>>>>>>>>>>> var/ipfire/ppp
>>>>>>>>>>>> var/ipfire/ppp/fake-resolv.conf
>>>>>>>>>>>> var/ipfire/ppp/settings
>>>>>>>>>>>> var/ipfire/private/cakey.pem
>>>>>>>>>>>> var/ipfire/proxy
>>>>>>>>>>>> var/ipfire/proxy/asnbl-helper.conf
>>>>>>>>>>>> var/ipfire/proxy/cachemgr.conf
>>>>>>>>>>>> var/ipfire/proxy/enable
>>>>>>>>>>>> var/ipfire/proxy/settings
>>>>>>>>>>>> var/ipfire/proxy/squid.conf
>>>>>>>>>>>> var/ipfire/qos/bin
>>>>>>>>>>>> var/ipfire/qos/classes
>>>>>>>>>>>> var/ipfire/qos/level7config
>>>>>>>>>>>> var/ipfire/qos/portconfig
>>>>>>>>>>>> var/ipfire/qos/settings
>>>>>>>>>>>> var/ipfire/qos/subclasses
>>>>>>>>>>>> var/ipfire/qos/tosconfig
>>>>>>>>>>>> var/ipfire/remote/enablessh
>>>>>>>>>>>> var/ipfire/remote/settings
>>>>>>>>>>>> var/ipfire/sensors/settings
>>>>>>>>>>>> var/ipfire/suricata/oinkmaster.conf
>>>>>>>>>>>> var/ipfire/suricata/oinkmaster-modify-sids.conf
>>>>>>>>>>>> var/ipfire/suricata/oinkmaster-provider-includes.conf
>>>>>>>>>>>> var/ipfire/suricata/providers-settings
>>>>>>>>>>>> var/ipfire/suricata/settings
>>>>>>>>>>>> var/ipfire/suricata/suricata-default-rules.yaml
>>>>>>>>>>>> var/ipfire/suricata/suricata-dns-servers.yaml
>>>>>>>>>>>> var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
>>>>>>>>>>>> var/ipfire/suricata/suricata-homenet.yaml
>>>>>>>>>>>> var/ipfire/suricata/suricata-http-ports.yaml
>>>>>>>>>>>> var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml
>>>>>>>>>>>> var/ipfire/suricata/suricata-used-providers.yaml
>>>>>>>>>>>> var/ipfire/time/
>>>>>>>>>>>> var/ipfire/time/counter.conf
>>>>>>>>>>>> var/ipfire/time/enable
>>>>>>>>>>>> var/ipfire/time/settime.conf
>>>>>>>>>>>> var/ipfire/time/settings
>>>>>>>>>>>> var/ipfire/upnp/settings
>>>>>>>>>>>> var/ipfire/urlfilter
>>>>>>>>>>>> var/ipfire/urlfilter/settings
>>>>>>>>>>>> var/ipfire/urlfilter/squidGuard.conf
>>>>>>>>>>>> var/ipfire/vpn
>>>>>>>>>>>> var/ipfire/vpn/config
>>>>>>>>>>>> var/ipfire/vpn/ipsec.conf
>>>>>>>>>>>> var/ipfire/vpn/settings
>>>>>>>>>>>> var/ipfire/wakeonlan/clients.conf
>>>>>>>>>>>> var/ipfire/wio/wio.conf
>>>>>>>>>>>> var/ipfire/wireless/config
>>>>>>>>>>>> var/ipfire/wireless/settings
>>>>>>>>>>>> var/lib/suricata
>>>>>>>>>>>> var/log/rrd/collectd
>>>>>>>>>>>> var/log/rrd/hddshutdown-md127.rrd
>>>>>>>>>>>> var/log/rrd/hddshutdown-sda.rrd
>>>>>>>>>>>> var/log/rrd/hddshutdown-sdb.rrd
>>>>>>>>>>>> var/log/rrd/hddtemp-md127.rrd
>>>>>>>>>>>> var/log/rrd/hddtemp-sda.rrd
>>>>>>>>>>>> var/log/rrd/hddtemp-sdb.rrd
>>>>>>>>>>>> var/log/rrd/wio
>>>>>>>>>>>> var/log/vnstat
>>>>>>>>>>>> var/tmp/idsrules-emerging.tar.gz
>>>>>>>>>>>> var/tmp/idsrules-sslbl_blacklist.rules
>>>>>>>>>>>> The following are in the previous list but not in this one:-
>>>>>>>>>>>> /
>>>>>>>>>>>> /etc/conntrackd/conntrackd.conf
>>>>>>>>>>>> /etc/ipsec.user-post.conf
>>>>>>>>>>>> /root/.gitconfig
>>>>>>>>>>>> /root/.ssh
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Adolf.
>>>>>>>>>>>> On 29/03/2022 16:01, Adolf Belka wrote:
>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 29/03/2022 15:36, Bernhard Bitsch wrote:
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Am 29.03.2022 um 15:11 schrieb Michael Tremer:
>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Could you please send the file listing to find out what is being included what shouldn’t?
>>>>>>>>>>>>> Unfortunately, as I stopped the backup continuing once it had reached 1.2GB, the file created was not able to be opened. Probably stopping the backup corrupted it in some way.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Just for easy quick handling, you can use
>>>>>>>>>>>>>> '/var/ipfire/backup/bin/backup.pl list' to show the files included.
>>>>>>>>>>>>> I gave this a go and here is the output from the command.
>>>>>>>>>>>>>
>>>>>>>>>>>>> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
>>>>>>>>>>>>> /
>>>>>>>>>>>>> /etc/conntrackd/conntrackd.conf
>>>>>>>>>>>>> /etc/group
>>>>>>>>>>>>> /etc/hosts
>>>>>>>>>>>>> /etc/hosts.allow
>>>>>>>>>>>>> /etc/hosts.deny
>>>>>>>>>>>>> /etc/httpd/server.crt
>>>>>>>>>>>>> /etc/httpd/server.csr
>>>>>>>>>>>>> /etc/httpd/server-ecdsa.crt
>>>>>>>>>>>>> /etc/httpd/server-ecdsa.csr
>>>>>>>>>>>>> /etc/httpd/server-ecdsa.key
>>>>>>>>>>>>> /etc/httpd/server.key
>>>>>>>>>>>>> /etc/ipsec.user.conf
>>>>>>>>>>>>> /etc/ipsec.user-post.conf
>>>>>>>>>>>>> /etc/ipsec.user.secrets
>>>>>>>>>>>>> /etc/logrotate.d
>>>>>>>>>>>>> /etc/passwd
>>>>>>>>>>>>> /etc/shadow
>>>>>>>>>>>>> /etc/squid/squid.conf.local
>>>>>>>>>>>>> /etc/squid/squid.conf.pre.local
>>>>>>>>>>>>> /etc/ssh/sshd_config
>>>>>>>>>>>>> /etc/ssh/ssh_host_ecdsa_key
>>>>>>>>>>>>> /etc/ssh/ssh_host_ecdsa_key.pub
>>>>>>>>>>>>> /etc/ssh/ssh_host_ed25519_key
>>>>>>>>>>>>> /etc/ssh/ssh_host_ed25519_key.pub
>>>>>>>>>>>>> /etc/ssh/ssh_host_rsa_key
>>>>>>>>>>>>> /etc/ssh/ssh_host_rsa_key.pub
>>>>>>>>>>>>> //etc/sudoers
>>>>>>>>>>>>> /etc/sysconfig/createfiles
>>>>>>>>>>>>> /etc/sysconfig/firewall.local
>>>>>>>>>>>>> /etc/sysconfig/lm_sensors
>>>>>>>>>>>>> /etc/sysconfig/modules
>>>>>>>>>>>>> /etc/sysconfig/ramdisk
>>>>>>>>>>>>> /etc/sysconfig/rc
>>>>>>>>>>>>> /etc/sysconfig/rc.local
>>>>>>>>>>>>> /etc/unbound
>>>>>>>>>>>>> //home/ahb
>>>>>>>>>>>>> /root/.bash_history
>>>>>>>>>>>>> /root/.gitconfig
>>>>>>>>>>>>> /root/.ssh
>>>>>>>>>>>>> /var/ipfire/accounting/settings.conf
>>>>>>>>>>>>> /var/ipfire/auth/users
>>>>>>>>>>>>> /var/ipfire/backup/addons/backup
>>>>>>>>>>>>> /var/ipfire/backup/exclude.user
>>>>>>>>>>>>> /var/ipfire/backup/include.user
>>>>>>>>>>>>> /var/ipfire/ca/cacert.pem
>>>>>>>>>>>>> /var/ipfire/captive/agb.txt
>>>>>>>>>>>>> /var/ipfire/captive/clients
>>>>>>>>>>>>> /var/ipfire/captive/coupons
>>>>>>>>>>>>> /var/ipfire/captive/logo.dat
>>>>>>>>>>>>> /var/ipfire/captive/settings
>>>>>>>>>>>>> /var/ipfire/captive/terms.txt
>>>>>>>>>>>>> /var/ipfire/captive/voucher_out
>>>>>>>>>>>>> /var/ipfire/certs/hostcert.pem
>>>>>>>>>>>>> /var/ipfire/certs/hostkey.pem
>>>>>>>>>>>>> /var/ipfire/certs/phoebevmipseccert.pem
>>>>>>>>>>>>> /var/ipfire/connscheduler/connscheduler.conf
>>>>>>>>>>>>> /var/ipfire/crls/cacrl.pem
>>>>>>>>>>>>> /var/ipfire/cups/cups-browsed.conf
>>>>>>>>>>>>> /var/ipfire/cups/subscriptions.conf
>>>>>>>>>>>>> /var/ipfire/ddns/config
>>>>>>>>>>>>> /var/ipfire/ddns/ddns.conf
>>>>>>>>>>>>> /var/ipfire/ddns/settings
>>>>>>>>>>>>> /var/ipfire/dhcp/advoptions
>>>>>>>>>>>>> /var/ipfire/dhcp/advoptions-list
>>>>>>>>>>>>> /var/ipfire/dhcpc/dhcpcd.conf
>>>>>>>>>>>>> /var/ipfire/dhcp/dhcpd.conf
>>>>>>>>>>>>> /var/ipfire/dhcp/dhcpd.conf.local
>>>>>>>>>>>>> /var/ipfire/dhcp/enable_blue
>>>>>>>>>>>>> /var/ipfire/dhcp/enable_green
>>>>>>>>>>>>> /var/ipfire/dhcp/fixleases
>>>>>>>>>>>>> /var/ipfire/dhcp/settings
>>>>>>>>>>>>> /var/ipfire/dma/auth.conf
>>>>>>>>>>>>> /var/ipfire/dma/dma.conf
>>>>>>>>>>>>> /var/ipfire/dma/mail.conf
>>>>>>>>>>>>> /var/ipfire/dns
>>>>>>>>>>>>> /var/ipfire/dnsforward/config
>>>>>>>>>>>>> /var/ipfire/dns/settings
>>>>>>>>>>>>> /var/ipfire/ethernet/aliases
>>>>>>>>>>>>> /var/ipfire/ethernet/settings
>>>>>>>>>>>>> /var/ipfire/ethernet/wireless
>>>>>>>>>>>>> /var/ipfire/extrahd/settings
>>>>>>>>>>>>> /var/ipfire/firewall
>>>>>>>>>>>>> /var/ipfire/firewall/config
>>>>>>>>>>>>> /var/ipfire/firewall/settings
>>>>>>>>>>>>> /var/ipfire/fwhosts
>>>>>>>>>>>>> /var/ipfire/isdn/settings
>>>>>>>>>>>>> /var/ipfire/logging/settings
>>>>>>>>>>>>> /var/ipfire/mac/settings
>>>>>>>>>>>>> /var/ipfire/main/firstsetup_ok
>>>>>>>>>>>>> /var/ipfire/main/gpl_accepted
>>>>>>>>>>>>> /var/ipfire/main/hostname.conf
>>>>>>>>>>>>> /var/ipfire/main/hosts
>>>>>>>>>>>>> /var/ipfire/main/manualpages
>>>>>>>>>>>>> /var/ipfire/main/routing
>>>>>>>>>>>>> /var/ipfire/main/security
>>>>>>>>>>>>> /var/ipfire/main/send_profile
>>>>>>>>>>>>> /var/ipfire/main/settings
>>>>>>>>>>>>> /var/ipfire/modem/settings
>>>>>>>>>>>>> /var/ipfire/optionsfw/settings
>>>>>>>>>>>>> /var/ipfire/ovpn
>>>>>>>>>>>>> /var/ipfire/ovpn/ccd.conf
>>>>>>>>>>>>> /var/ipfire/ovpn/collectd.vpn
>>>>>>>>>>>>> /var/ipfire/ovpn/enable
>>>>>>>>>>>>> /var/ipfire/ovpn/server.conf
>>>>>>>>>>>>> /var/ipfire/ovpn/settings
>>>>>>>>>>>>> /var/ipfire/pakfire/settings
>>>>>>>>>>>>> /var/ipfire/ppp
>>>>>>>>>>>>> /var/ipfire/ppp/fake-resolv.conf
>>>>>>>>>>>>> /var/ipfire/ppp/settings
>>>>>>>>>>>>> /var/ipfire/private/cakey.pem
>>>>>>>>>>>>> /var/ipfire/proxy
>>>>>>>>>>>>> /var/ipfire/proxy/asnbl-helper.conf
>>>>>>>>>>>>> /var/ipfire/proxy/cachemgr.conf
>>>>>>>>>>>>> /var/ipfire/proxy/enable
>>>>>>>>>>>>> /var/ipfire/proxy/settings
>>>>>>>>>>>>> /var/ipfire/proxy/squid.conf
>>>>>>>>>>>>> /var/ipfire/qos/bin
>>>>>>>>>>>>> /var/ipfire/qos/bin/qos.sh
>>>>>>>>>>>>> /var/ipfire/qos/classes
>>>>>>>>>>>>> /var/ipfire/qos/level7config
>>>>>>>>>>>>> /var/ipfire/qos/portconfig
>>>>>>>>>>>>> /var/ipfire/qos/settings
>>>>>>>>>>>>> /var/ipfire/qos/subclasses
>>>>>>>>>>>>> /var/ipfire/qos/tosconfig
>>>>>>>>>>>>> /var/ipfire/remote/enablessh
>>>>>>>>>>>>> /var/ipfire/remote/settings
>>>>>>>>>>>>> /var/ipfire/sensors/settings
>>>>>>>>>>>>> /var/ipfire/suricata/oinkmaster.conf
>>>>>>>>>>>>> /var/ipfire/suricata/oinkmaster-modify-sids.conf
>>>>>>>>>>>>> /var/ipfire/suricata/oinkmaster-provider-includes.conf
>>>>>>>>>>>>> /var/ipfire/suricata/providers-settings
>>>>>>>>>>>>> /var/ipfire/suricata/settings
>>>>>>>>>>>>> /var/ipfire/suricata/suricata-default-rules.yaml
>>>>>>>>>>>>> /var/ipfire/suricata/suricata-dns-servers.yaml
>>>>>>>>>>>>> /var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
>>>>>>>>>>>>> /var/ipfire/suricata/suricata-homenet.yaml
>>>>>>>>>>>>> /var/ipfire/suricata/suricata-http-ports.yaml
>>>>>>>>>>>>> /var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml
>>>>>>>>>>>>> /var/ipfire/suricata/suricata-used-providers.yaml
>>>>>>>>>>>>> /var/ipfire/time/
>>>>>>>>>>>>> /var/ipfire/time/counter.conf
>>>>>>>>>>>>> /var/ipfire/time/enable
>>>>>>>>>>>>> /var/ipfire/time/settime.conf
>>>>>>>>>>>>> /var/ipfire/time/settings
>>>>>>>>>>>>> /var/ipfire/upnp/settings
>>>>>>>>>>>>> /var/ipfire/urlfilter
>>>>>>>>>>>>> /var/ipfire/urlfilter/settings
>>>>>>>>>>>>> /var/ipfire/urlfilter/squidGuard.conf
>>>>>>>>>>>>> /var/ipfire/vpn
>>>>>>>>>>>>> /var/ipfire/vpn/config
>>>>>>>>>>>>> /var/ipfire/vpn/ipsec.conf
>>>>>>>>>>>>> /var/ipfire/vpn/settings
>>>>>>>>>>>>> /var/ipfire/wakeonlan/clients.conf
>>>>>>>>>>>>> /var/ipfire/wio/wio.conf
>>>>>>>>>>>>> /var/ipfire/wireless/config
>>>>>>>>>>>>> /var/ipfire/wireless/settings
>>>>>>>>>>>>> /var/lib/suricata
>>>>>>>>>>>>> /var/log/rrd/collectd
>>>>>>>>>>>>> /var/log/rrd/hddshutdown-md127.rrd
>>>>>>>>>>>>> /var/log/rrd/hddshutdown-sda.rrd
>>>>>>>>>>>>> /var/log/rrd/hddshutdown-sdb.rrd
>>>>>>>>>>>>> /var/log/rrd/hddtemp-md127.rrd
>>>>>>>>>>>>> /var/log/rrd/hddtemp-sda.rrd
>>>>>>>>>>>>> /var/log/rrd/hddtemp-sdb.rrd
>>>>>>>>>>>>> /var/log/rrd/wio
>>>>>>>>>>>>> /var/log/vnstat
>>>>>>>>>>>>> /var/tmp/idsrules-emerging.tar.gz
>>>>>>>>>>>>> /var/tmp/idsrules-sslbl_blacklist.rules
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Bernhard
>>>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 29 Mar 2022, at 14:10, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Just tried this patch out on my vm testbed system and it still doesn't work for me. The backup file had got to 1.3GB when I deleted the backup file as it was still growing. The normal correct backup file on that vm machine is around 7MB
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The above was the case for both running it from the WUI or from the command line from my unprivileged user using sudo backupctrl exclude
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 29/03/2022 14:27, Michael Tremer wrote:
>>>>>>>>>>>>>>>>> This patch fixes globbing expansion in the backup include file list
>>>>>>>>>>>>>>>>> which got broken in c7e0d73e7cfd7be95db9d0a5f3392b8241813d5b.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>> config/backup/backup.pl | 8 ++++----
>>>>>>>>>>>>>>>>> 1 file changed, 4 insertions(+), 4 deletions(-)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> diff --git a/config/backup/backup.pl b/config/backup/backup.pl
>>>>>>>>>>>>>>>>> index a2337cf23..6f9295e94 100644
>>>>>>>>>>>>>>>>> --- a/config/backup/backup.pl
>>>>>>>>>>>>>>>>> +++ b/config/backup/backup.pl
>>>>>>>>>>>>>>>>> @@ -19,6 +19,8 @@
>>>>>>>>>>>>>>>>> # #
>>>>>>>>>>>>>>>>> ###############################################################################
>>>>>>>>>>>>>>>>> +shopt -s nullglob
>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>> NOW="$(date "+%Y-%m-%d-%H:%M")"
>>>>>>>>>>>>>>>>> list_addons() {
>>>>>>>>>>>>>>>>> @@ -38,10 +40,8 @@ process_includes() {
>>>>>>>>>>>>>>>>> for include in $@; do
>>>>>>>>>>>>>>>>> local file
>>>>>>>>>>>>>>>>> while read -r file; do
>>>>>>>>>>>>>>>>> -  for file in ${file}; do
>>>>>>>>>>>>>>>>> -  if [ -e "/${file}" ]; then
>>>>>>>>>>>>>>>>> -  echo "${file}"
>>>>>>>>>>>>>>>>> -  fi
>>>>>>>>>>>>>>>>> +  for file in /${file}; do
>>>>>>>>>>>>>>>>> +  echo "${file}"
>>>>>>>>>>>>>>>>> done
>>>>>>>>>>>>>>>>> done < "${include}"
>>>>>>>>>>>>>>>>> done | sort -u
>

Jon Murphy March 31, 2022, 6:22 p.m. UTC | #24

Adolf,

Might this issue be part of the overall issue?  I am wondering if it was tested in the special release build?

https://bugzilla.ipfire.org/show_bug.cgi?id=12811 <https://bugzilla.ipfire.org/show_bug.cgi?id=12811>

Jon

> On Mar 31, 2022, at 5:16 AM, Adolf Belka <adolf.belka@ipfire.org> wrote:
> 
> Hi Michael,
> 
> Update on the testing. I realised that I had done the test on a CU175 version where I had modified the backup.pl code with pushd/popd.
> I wouldn't expect that to give a problem but to confirm I created another CU165 clone with the backup.pl code in original form as most standard users would have it.
> Re-ran the test and everything went correctly the same as my previous test.
> 
> Just for security I also did a fresh install of CU166 from the latest master nightly.
> That also worked fine. The backup had all the directories as expected.
> 
> Regards,
> Adolf
> 
> On 31/03/2022 09:46, Michael Tremer wrote:
>> Hello Adolf,
>> Brilliant. Thank you very much for testing this in detail.
>> Best,
>> -Michael
>>> On 31 Mar 2022, at 08:41, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>> 
>>> Hi Michael,
>>> 
>>> On 30/03/2022 23:20, Adolf Belka wrote:
>>>> Hi,
>>>> 
>>>> On 30/03/2022 17:05, Michael Tremer wrote:
>>>>> Hello Bernhard,
>>>>> 
>>>>> You are obviously right. My brain is absolute mush these days and doesn’t allow me to function.
>>>>> 
>>>>> I will correct this by the end of the day.
>>>>> 
>>>>> -Michael
>>>>> 
>>>>>> On 30 Mar 2022, at 15:38, Bernhard Bitsch <bbitsch@ipfire.org> wrote:
>>>>>> 
>>>>>> Hello all,
>>>>>> 
>>>>>> 
>>>>>> Am 30.03.2022 um 16:06 schrieb Michael Tremer:
>>>>>>> Hello everyone,
>>>>>>> Thank you very much for helping me test this.
>>>>>>> I decided that we will have to create an emergency core update for this, because it will otherwise break people’s system. We also have a vulnerability in zlib which is being handed around by the press, so that has helped me to make a decision.
>>>>>> 
>>>>>> I think this is a good solution. Shall we add a warning to backup wiki page? Because in the moment a backup -> reinstall -> restore process leaves an incomplete system.
>>>>>> 
>>>>>>> I also pushed two more patches. One filters out any empty lines because they have quite a damaging effect and it happens easily that they are added to the include list. Secondly, I brought back the existence check which I thought was only there for the globbing mechanism, but also had the small side-effect of filtering out the file list early on which helps keeping tar happy.
>>>>>>> Please review those changes and please install the new c166 from testing as soon as the build has finished (I am not sure if you are all on the nightly-builds list to see any notifications).
>>>> I am not on the notifications list but I will check first thing tomorrow and when the latest has been updated I will do a test install and evaluation of the backup and provide confirmation of how it functions.
>>>> 
>>> As CU166 has been moved to master I took a clone of my CU165 vm and then did a CU on Testing with it rather than doing a fresh install. If you would also want a test of a fresh install done I can also do that.
>>> 
>>> The version I tested was IPFire 2.27 (x86_64) - Core Update 166 Development Build: master/8f696f60
>>> 
>>> Anyway the backup is working fine again. All the directories are present. The size of the backup is now very similar to previously also confirming all contents should be good.
>>> 
>>> I also added back in a couple of blank lines into my include.user file and the backup worked without any problems.
>>> 
>>> I also did a restore of the backup and again as far as I can tell with a quick check through on my testbed, everything is fine.
>>> 
>>> So everything looks good to me.
>>> 
>>> Regards,
>>> 
>>> Adolf.
>>> 
>>>> Regards,
>>>> Adolf.
>>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=4f0e7f24f293ca10096e616010cfd4417afc157c
>>>>>> The test on empty string should be in the 'while read -r file' loop. The empty lines are in the include files. The outer for loop iterates over the file list.
>>>>>> 
>>>>>> Regards,
>>>>>> Bernhard
>>>>>> 
>>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=b275771fdd10ac7116e5d400dc10d3148df8ac27
>>>>>>> Best,
>>>>>>> -Michael
>>>>>>>> On 29 Mar 2022, at 20:21, Bernhard Bitsch <bbitsch@ipfire.org> wrote:
>>>>>>>> 
>>>>>>>> Just checked the blank line problem.
>>>>>>>> Pattern is empty , file name list is '/' --> no names produced ( with existence check )
>>>>>>>> Pattern is empty --> produced file name '/' ( without -e check )
>>>>>>>> 
>>>>>>>> Am 29.03.2022 um 20:36 schrieb Bernhard Bitsch:
>>>>>>>>> Hi all,
>>>>>>>>> Am 29.03.2022 um 17:23 schrieb Adolf Belka:
>>>>>>>>>> Hi All,
>>>>>>>>>> 
>>>>>>>>>> On 29/03/2022 16:22, Michael Tremer wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> On 29 Mar 2022, at 15:22, Bernhard Bitsch <bbitsch@ipfire.org> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> Looks like your include.user file contains absolute paths.
>>>>>>>>>> That was not the problem but I have corrected that. I would have expected that when the main include and exclude files were made relative that the update script would also have checked for any entries in the include.user and exclude.user files but it looks like it didn't.th
>>>>>>>>> According to the source they are chosen.
>>>>>>>>>>> 
>>>>>>>>>>> Or an empty line maybe?
>>>>>>>>>> That turned out to be the problem. There was an empty line after the two lines I had added. You only saw it if you scrolled the pointer down the file in the editor.
>>>>>>>>> Interesting. A blank line should produce nothing.
>>>>>>>>>> 
>>>>>>>>>> Removing that blank line removed the line only with a / and the backup then successfully ran and created a backup file containing all the directories expected.
>>>>>>>>>> 
>>>>>>>>>> The only thing I found is that there was the following tar error message
>>>>>>>>>> 
>>>>>>>>>> tar: Exiting with failure status due to previous errors
>>>>>>>>>> 
>>>>>>>>>> The only thing I found in the backup output was several lines such as
>>>>>>>>>> 
>>>>>>>>>> tar: /etc/squid/squid.conf.local: Cannot stat: No such file or directory
>>>>>>>>>> tar: /etc/squid/squid.conf.pre.local: Cannot stat: No such file or directory
>>>>>>>>>> 
>>>>>>>>>> There were seven lines in total like this and the three I checked were files that did not exist on my system. I presume that these are files that should be backed up if present but don't have to be present so that the error messages are nothing to worry about.
>>>>>>>>>> There were no error messages when doing the backup via the WUI.
>>>>>>>>>> 
>>>>>>>>> Good find. ;)
>>>>>>>>> Michael's patch removes the test of existence, globbing produces existent file names only. But I suppose there are several full names of files, not existing in each implementation.
>>>>>>>>> Conclusion: Repair the file globbing by the shopt and do the existence check. So all existent files are chosen, that are listed in the include files.
>>>>>>>>> Regards,
>>>>>>>>> Bernhard
>>>>>>>>>> The blank line was only on my vm testbed system but as I clone this for any testing activities I will have to check all of my existing vm's to fix this.
>>>>>>>>>> 
>>>>>>>>>> Glad we resolved this simply. Sorry for the anxiety.
>>>>>>>>>> 
>>>>>>>>>> Regards,
>>>>>>>>>> 
>>>>>>>>>> Adolf.
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> Am 29.03.2022 um 16:14 schrieb Adolf Belka:
>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>> This is what I get with the pushd/popd version in place on the same system
>>>>>>>>>>>>> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
>>>>>>>>>>>>> etc/group
>>>>>>>>>>>>> etc/hosts
>>>>>>>>>>>>> etc/hosts.allow
>>>>>>>>>>>>> etc/hosts.deny
>>>>>>>>>>>>> etc/httpd/server.crt
>>>>>>>>>>>>> etc/httpd/server.csr
>>>>>>>>>>>>> etc/httpd/server-ecdsa.crt
>>>>>>>>>>>>> etc/httpd/server-ecdsa.csr
>>>>>>>>>>>>> etc/httpd/server-ecdsa.key
>>>>>>>>>>>>> etc/httpd/server.key
>>>>>>>>>>>>> etc/ipsec.user.conf
>>>>>>>>>>>>> etc/ipsec.user.secrets
>>>>>>>>>>>>> etc/logrotate.d
>>>>>>>>>>>>> etc/passwd
>>>>>>>>>>>>> etc/shadow
>>>>>>>>>>>>> etc/ssh/sshd_config
>>>>>>>>>>>>> etc/ssh/ssh_host_ecdsa_key
>>>>>>>>>>>>> etc/ssh/ssh_host_ecdsa_key.pub
>>>>>>>>>>>>> etc/ssh/ssh_host_ed25519_key
>>>>>>>>>>>>> etc/ssh/ssh_host_ed25519_key.pub
>>>>>>>>>>>>> etc/ssh/ssh_host_rsa_key
>>>>>>>>>>>>> etc/ssh/ssh_host_rsa_key.pub
>>>>>>>>>>>>> /etc/sudoers
>>>>>>>>>>>>> etc/sysconfig/createfiles
>>>>>>>>>>>>> etc/sysconfig/firewall.local
>>>>>>>>>>>>> etc/sysconfig/lm_sensors
>>>>>>>>>>>>> etc/sysconfig/modules
>>>>>>>>>>>>> etc/sysconfig/ramdisk
>>>>>>>>>>>>> etc/sysconfig/rc
>>>>>>>>>>>>> etc/sysconfig/rc.local
>>>>>>>>>>>>> etc/unbound
>>>>>>>>>>>>> /home/ahb
>>>>>>>>>>>>> root/.bash_history
>>>>>>>>>>>>> var/ipfire/accounting/settings.conf
>>>>>>>>>>>>> var/ipfire/auth/users
>>>>>>>>>>>>> var/ipfire/backup/addons/backup
>>>>>>>>>>>>> var/ipfire/backup/exclude.user
>>>>>>>>>>>>> var/ipfire/backup/include.user
>>>>>>>>>>>>> var/ipfire/ca/cacert.pem
>>>>>>>>>>>>> var/ipfire/captive/agb.txt
>>>>>>>>>>>>> var/ipfire/captive/clients
>>>>>>>>>>>>> var/ipfire/captive/coupons
>>>>>>>>>>>>> var/ipfire/captive/logo.dat
>>>>>>>>>>>>> var/ipfire/captive/settings
>>>>>>>>>>>>> var/ipfire/captive/terms.txt
>>>>>>>>>>>>> var/ipfire/captive/voucher_out
>>>>>>>>>>>>> var/ipfire/certs/hostcert.pem
>>>>>>>>>>>>> var/ipfire/certs/hostkey.pem
>>>>>>>>>>>>> var/ipfire/certs/phoebevmipseccert.pem
>>>>>>>>>>>>> var/ipfire/connscheduler/connscheduler.conf
>>>>>>>>>>>>> var/ipfire/crls/cacrl.pem
>>>>>>>>>>>>> var/ipfire/cups/cups-browsed.conf
>>>>>>>>>>>>> var/ipfire/cups/subscriptions.conf
>>>>>>>>>>>>> var/ipfire/ddns/config
>>>>>>>>>>>>> var/ipfire/ddns/ddns.conf
>>>>>>>>>>>>> var/ipfire/ddns/settings
>>>>>>>>>>>>> var/ipfire/dhcp/advoptions
>>>>>>>>>>>>> var/ipfire/dhcp/advoptions-list
>>>>>>>>>>>>> var/ipfire/dhcpc/dhcpcd.conf
>>>>>>>>>>>>> var/ipfire/dhcp/dhcpd.conf
>>>>>>>>>>>>> var/ipfire/dhcp/dhcpd.conf.local
>>>>>>>>>>>>> var/ipfire/dhcp/enable_blue
>>>>>>>>>>>>> var/ipfire/dhcp/enable_green
>>>>>>>>>>>>> var/ipfire/dhcp/fixleases
>>>>>>>>>>>>> var/ipfire/dhcp/settings
>>>>>>>>>>>>> var/ipfire/dma/auth.conf
>>>>>>>>>>>>> var/ipfire/dma/dma.conf
>>>>>>>>>>>>> var/ipfire/dma/mail.conf
>>>>>>>>>>>>> var/ipfire/dns
>>>>>>>>>>>>> var/ipfire/dnsforward/config
>>>>>>>>>>>>> var/ipfire/dns/settings
>>>>>>>>>>>>> var/ipfire/ethernet/aliases
>>>>>>>>>>>>> var/ipfire/ethernet/settings
>>>>>>>>>>>>> var/ipfire/ethernet/wireless
>>>>>>>>>>>>> var/ipfire/extrahd/settings
>>>>>>>>>>>>> var/ipfire/firewall
>>>>>>>>>>>>> var/ipfire/firewall/config
>>>>>>>>>>>>> var/ipfire/firewall/settings
>>>>>>>>>>>>> var/ipfire/fwhosts
>>>>>>>>>>>>> var/ipfire/isdn/settings
>>>>>>>>>>>>> var/ipfire/logging/settings
>>>>>>>>>>>>> var/ipfire/mac/settings
>>>>>>>>>>>>> var/ipfire/main/firstsetup_ok
>>>>>>>>>>>>> var/ipfire/main/gpl_accepted
>>>>>>>>>>>>> var/ipfire/main/hostname.conf
>>>>>>>>>>>>> var/ipfire/main/hosts
>>>>>>>>>>>>> var/ipfire/main/manualpages
>>>>>>>>>>>>> var/ipfire/main/routing
>>>>>>>>>>>>> var/ipfire/main/security
>>>>>>>>>>>>> var/ipfire/main/send_profile
>>>>>>>>>>>>> var/ipfire/main/settings
>>>>>>>>>>>>> var/ipfire/modem/settings
>>>>>>>>>>>>> var/ipfire/optionsfw/settings
>>>>>>>>>>>>> var/ipfire/ovpn
>>>>>>>>>>>>> var/ipfire/ovpn/ccd.conf
>>>>>>>>>>>>> var/ipfire/ovpn/collectd.vpn
>>>>>>>>>>>>> var/ipfire/ovpn/enable
>>>>>>>>>>>>> var/ipfire/ovpn/server.conf
>>>>>>>>>>>>> var/ipfire/ovpn/settings
>>>>>>>>>>>>> var/ipfire/pakfire/settings
>>>>>>>>>>>>> var/ipfire/ppp
>>>>>>>>>>>>> var/ipfire/ppp/fake-resolv.conf
>>>>>>>>>>>>> var/ipfire/ppp/settings
>>>>>>>>>>>>> var/ipfire/private/cakey.pem
>>>>>>>>>>>>> var/ipfire/proxy
>>>>>>>>>>>>> var/ipfire/proxy/asnbl-helper.conf
>>>>>>>>>>>>> var/ipfire/proxy/cachemgr.conf
>>>>>>>>>>>>> var/ipfire/proxy/enable
>>>>>>>>>>>>> var/ipfire/proxy/settings
>>>>>>>>>>>>> var/ipfire/proxy/squid.conf
>>>>>>>>>>>>> var/ipfire/qos/bin
>>>>>>>>>>>>> var/ipfire/qos/classes
>>>>>>>>>>>>> var/ipfire/qos/level7config
>>>>>>>>>>>>> var/ipfire/qos/portconfig
>>>>>>>>>>>>> var/ipfire/qos/settings
>>>>>>>>>>>>> var/ipfire/qos/subclasses
>>>>>>>>>>>>> var/ipfire/qos/tosconfig
>>>>>>>>>>>>> var/ipfire/remote/enablessh
>>>>>>>>>>>>> var/ipfire/remote/settings
>>>>>>>>>>>>> var/ipfire/sensors/settings
>>>>>>>>>>>>> var/ipfire/suricata/oinkmaster.conf
>>>>>>>>>>>>> var/ipfire/suricata/oinkmaster-modify-sids.conf
>>>>>>>>>>>>> var/ipfire/suricata/oinkmaster-provider-includes.conf
>>>>>>>>>>>>> var/ipfire/suricata/providers-settings
>>>>>>>>>>>>> var/ipfire/suricata/settings
>>>>>>>>>>>>> var/ipfire/suricata/suricata-default-rules.yaml
>>>>>>>>>>>>> var/ipfire/suricata/suricata-dns-servers.yaml
>>>>>>>>>>>>> var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
>>>>>>>>>>>>> var/ipfire/suricata/suricata-homenet.yaml
>>>>>>>>>>>>> var/ipfire/suricata/suricata-http-ports.yaml
>>>>>>>>>>>>> var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml
>>>>>>>>>>>>> var/ipfire/suricata/suricata-used-providers.yaml
>>>>>>>>>>>>> var/ipfire/time/
>>>>>>>>>>>>> var/ipfire/time/counter.conf
>>>>>>>>>>>>> var/ipfire/time/enable
>>>>>>>>>>>>> var/ipfire/time/settime.conf
>>>>>>>>>>>>> var/ipfire/time/settings
>>>>>>>>>>>>> var/ipfire/upnp/settings
>>>>>>>>>>>>> var/ipfire/urlfilter
>>>>>>>>>>>>> var/ipfire/urlfilter/settings
>>>>>>>>>>>>> var/ipfire/urlfilter/squidGuard.conf
>>>>>>>>>>>>> var/ipfire/vpn
>>>>>>>>>>>>> var/ipfire/vpn/config
>>>>>>>>>>>>> var/ipfire/vpn/ipsec.conf
>>>>>>>>>>>>> var/ipfire/vpn/settings
>>>>>>>>>>>>> var/ipfire/wakeonlan/clients.conf
>>>>>>>>>>>>> var/ipfire/wio/wio.conf
>>>>>>>>>>>>> var/ipfire/wireless/config
>>>>>>>>>>>>> var/ipfire/wireless/settings
>>>>>>>>>>>>> var/lib/suricata
>>>>>>>>>>>>> var/log/rrd/collectd
>>>>>>>>>>>>> var/log/rrd/hddshutdown-md127.rrd
>>>>>>>>>>>>> var/log/rrd/hddshutdown-sda.rrd
>>>>>>>>>>>>> var/log/rrd/hddshutdown-sdb.rrd
>>>>>>>>>>>>> var/log/rrd/hddtemp-md127.rrd
>>>>>>>>>>>>> var/log/rrd/hddtemp-sda.rrd
>>>>>>>>>>>>> var/log/rrd/hddtemp-sdb.rrd
>>>>>>>>>>>>> var/log/rrd/wio
>>>>>>>>>>>>> var/log/vnstat
>>>>>>>>>>>>> var/tmp/idsrules-emerging.tar.gz
>>>>>>>>>>>>> var/tmp/idsrules-sslbl_blacklist.rules
>>>>>>>>>>>>> The following are in the previous list but not in this one:-
>>>>>>>>>>>>> /
>>>>>>>>>>>>> /etc/conntrackd/conntrackd.conf
>>>>>>>>>>>>> /etc/ipsec.user-post.conf
>>>>>>>>>>>>> /root/.gitconfig
>>>>>>>>>>>>> /root/.ssh
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>> On 29/03/2022 16:01, Adolf Belka wrote:
>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On 29/03/2022 15:36, Bernhard Bitsch wrote:
>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Am 29.03.2022 um 15:11 schrieb Michael Tremer:
>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Could you please send the file listing to find out what is being included what shouldn’t?
>>>>>>>>>>>>>> Unfortunately, as I stopped the backup continuing once it had reached 1.2GB, the file created was not able to be opened. Probably stopping the backup corrupted it in some way.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Just for easy quick handling, you can use
>>>>>>>>>>>>>>> '/var/ipfire/backup/bin/backup.pl list' to show the files included.
>>>>>>>>>>>>>> I gave this a go and here is the output from the command.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
>>>>>>>>>>>>>> /
>>>>>>>>>>>>>> /etc/conntrackd/conntrackd.conf
>>>>>>>>>>>>>> /etc/group
>>>>>>>>>>>>>> /etc/hosts
>>>>>>>>>>>>>> /etc/hosts.allow
>>>>>>>>>>>>>> /etc/hosts.deny
>>>>>>>>>>>>>> /etc/httpd/server.crt
>>>>>>>>>>>>>> /etc/httpd/server.csr
>>>>>>>>>>>>>> /etc/httpd/server-ecdsa.crt
>>>>>>>>>>>>>> /etc/httpd/server-ecdsa.csr
>>>>>>>>>>>>>> /etc/httpd/server-ecdsa.key
>>>>>>>>>>>>>> /etc/httpd/server.key
>>>>>>>>>>>>>> /etc/ipsec.user.conf
>>>>>>>>>>>>>> /etc/ipsec.user-post.conf
>>>>>>>>>>>>>> /etc/ipsec.user.secrets
>>>>>>>>>>>>>> /etc/logrotate.d
>>>>>>>>>>>>>> /etc/passwd
>>>>>>>>>>>>>> /etc/shadow
>>>>>>>>>>>>>> /etc/squid/squid.conf.local
>>>>>>>>>>>>>> /etc/squid/squid.conf.pre.local
>>>>>>>>>>>>>> /etc/ssh/sshd_config
>>>>>>>>>>>>>> /etc/ssh/ssh_host_ecdsa_key
>>>>>>>>>>>>>> /etc/ssh/ssh_host_ecdsa_key.pub
>>>>>>>>>>>>>> /etc/ssh/ssh_host_ed25519_key
>>>>>>>>>>>>>> /etc/ssh/ssh_host_ed25519_key.pub
>>>>>>>>>>>>>> /etc/ssh/ssh_host_rsa_key
>>>>>>>>>>>>>> /etc/ssh/ssh_host_rsa_key.pub
>>>>>>>>>>>>>> //etc/sudoers
>>>>>>>>>>>>>> /etc/sysconfig/createfiles
>>>>>>>>>>>>>> /etc/sysconfig/firewall.local
>>>>>>>>>>>>>> /etc/sysconfig/lm_sensors
>>>>>>>>>>>>>> /etc/sysconfig/modules
>>>>>>>>>>>>>> /etc/sysconfig/ramdisk
>>>>>>>>>>>>>> /etc/sysconfig/rc
>>>>>>>>>>>>>> /etc/sysconfig/rc.local
>>>>>>>>>>>>>> /etc/unbound
>>>>>>>>>>>>>> //home/ahb
>>>>>>>>>>>>>> /root/.bash_history
>>>>>>>>>>>>>> /root/.gitconfig
>>>>>>>>>>>>>> /root/.ssh
>>>>>>>>>>>>>> /var/ipfire/accounting/settings.conf
>>>>>>>>>>>>>> /var/ipfire/auth/users
>>>>>>>>>>>>>> /var/ipfire/backup/addons/backup
>>>>>>>>>>>>>> /var/ipfire/backup/exclude.user
>>>>>>>>>>>>>> /var/ipfire/backup/include.user
>>>>>>>>>>>>>> /var/ipfire/ca/cacert.pem
>>>>>>>>>>>>>> /var/ipfire/captive/agb.txt
>>>>>>>>>>>>>> /var/ipfire/captive/clients
>>>>>>>>>>>>>> /var/ipfire/captive/coupons
>>>>>>>>>>>>>> /var/ipfire/captive/logo.dat
>>>>>>>>>>>>>> /var/ipfire/captive/settings
>>>>>>>>>>>>>> /var/ipfire/captive/terms.txt
>>>>>>>>>>>>>> /var/ipfire/captive/voucher_out
>>>>>>>>>>>>>> /var/ipfire/certs/hostcert.pem
>>>>>>>>>>>>>> /var/ipfire/certs/hostkey.pem
>>>>>>>>>>>>>> /var/ipfire/certs/phoebevmipseccert.pem
>>>>>>>>>>>>>> /var/ipfire/connscheduler/connscheduler.conf
>>>>>>>>>>>>>> /var/ipfire/crls/cacrl.pem
>>>>>>>>>>>>>> /var/ipfire/cups/cups-browsed.conf
>>>>>>>>>>>>>> /var/ipfire/cups/subscriptions.conf
>>>>>>>>>>>>>> /var/ipfire/ddns/config
>>>>>>>>>>>>>> /var/ipfire/ddns/ddns.conf
>>>>>>>>>>>>>> /var/ipfire/ddns/settings
>>>>>>>>>>>>>> /var/ipfire/dhcp/advoptions
>>>>>>>>>>>>>> /var/ipfire/dhcp/advoptions-list
>>>>>>>>>>>>>> /var/ipfire/dhcpc/dhcpcd.conf
>>>>>>>>>>>>>> /var/ipfire/dhcp/dhcpd.conf
>>>>>>>>>>>>>> /var/ipfire/dhcp/dhcpd.conf.local
>>>>>>>>>>>>>> /var/ipfire/dhcp/enable_blue
>>>>>>>>>>>>>> /var/ipfire/dhcp/enable_green
>>>>>>>>>>>>>> /var/ipfire/dhcp/fixleases
>>>>>>>>>>>>>> /var/ipfire/dhcp/settings
>>>>>>>>>>>>>> /var/ipfire/dma/auth.conf
>>>>>>>>>>>>>> /var/ipfire/dma/dma.conf
>>>>>>>>>>>>>> /var/ipfire/dma/mail.conf
>>>>>>>>>>>>>> /var/ipfire/dns
>>>>>>>>>>>>>> /var/ipfire/dnsforward/config
>>>>>>>>>>>>>> /var/ipfire/dns/settings
>>>>>>>>>>>>>> /var/ipfire/ethernet/aliases
>>>>>>>>>>>>>> /var/ipfire/ethernet/settings
>>>>>>>>>>>>>> /var/ipfire/ethernet/wireless
>>>>>>>>>>>>>> /var/ipfire/extrahd/settings
>>>>>>>>>>>>>> /var/ipfire/firewall
>>>>>>>>>>>>>> /var/ipfire/firewall/config
>>>>>>>>>>>>>> /var/ipfire/firewall/settings
>>>>>>>>>>>>>> /var/ipfire/fwhosts
>>>>>>>>>>>>>> /var/ipfire/isdn/settings
>>>>>>>>>>>>>> /var/ipfire/logging/settings
>>>>>>>>>>>>>> /var/ipfire/mac/settings
>>>>>>>>>>>>>> /var/ipfire/main/firstsetup_ok
>>>>>>>>>>>>>> /var/ipfire/main/gpl_accepted
>>>>>>>>>>>>>> /var/ipfire/main/hostname.conf
>>>>>>>>>>>>>> /var/ipfire/main/hosts
>>>>>>>>>>>>>> /var/ipfire/main/manualpages
>>>>>>>>>>>>>> /var/ipfire/main/routing
>>>>>>>>>>>>>> /var/ipfire/main/security
>>>>>>>>>>>>>> /var/ipfire/main/send_profile
>>>>>>>>>>>>>> /var/ipfire/main/settings
>>>>>>>>>>>>>> /var/ipfire/modem/settings
>>>>>>>>>>>>>> /var/ipfire/optionsfw/settings
>>>>>>>>>>>>>> /var/ipfire/ovpn
>>>>>>>>>>>>>> /var/ipfire/ovpn/ccd.conf
>>>>>>>>>>>>>> /var/ipfire/ovpn/collectd.vpn
>>>>>>>>>>>>>> /var/ipfire/ovpn/enable
>>>>>>>>>>>>>> /var/ipfire/ovpn/server.conf
>>>>>>>>>>>>>> /var/ipfire/ovpn/settings
>>>>>>>>>>>>>> /var/ipfire/pakfire/settings
>>>>>>>>>>>>>> /var/ipfire/ppp
>>>>>>>>>>>>>> /var/ipfire/ppp/fake-resolv.conf
>>>>>>>>>>>>>> /var/ipfire/ppp/settings
>>>>>>>>>>>>>> /var/ipfire/private/cakey.pem
>>>>>>>>>>>>>> /var/ipfire/proxy
>>>>>>>>>>>>>> /var/ipfire/proxy/asnbl-helper.conf
>>>>>>>>>>>>>> /var/ipfire/proxy/cachemgr.conf
>>>>>>>>>>>>>> /var/ipfire/proxy/enable
>>>>>>>>>>>>>> /var/ipfire/proxy/settings
>>>>>>>>>>>>>> /var/ipfire/proxy/squid.conf
>>>>>>>>>>>>>> /var/ipfire/qos/bin
>>>>>>>>>>>>>> /var/ipfire/qos/bin/qos.sh
>>>>>>>>>>>>>> /var/ipfire/qos/classes
>>>>>>>>>>>>>> /var/ipfire/qos/level7config
>>>>>>>>>>>>>> /var/ipfire/qos/portconfig
>>>>>>>>>>>>>> /var/ipfire/qos/settings
>>>>>>>>>>>>>> /var/ipfire/qos/subclasses
>>>>>>>>>>>>>> /var/ipfire/qos/tosconfig
>>>>>>>>>>>>>> /var/ipfire/remote/enablessh
>>>>>>>>>>>>>> /var/ipfire/remote/settings
>>>>>>>>>>>>>> /var/ipfire/sensors/settings
>>>>>>>>>>>>>> /var/ipfire/suricata/oinkmaster.conf
>>>>>>>>>>>>>> /var/ipfire/suricata/oinkmaster-modify-sids.conf
>>>>>>>>>>>>>> /var/ipfire/suricata/oinkmaster-provider-includes.conf
>>>>>>>>>>>>>> /var/ipfire/suricata/providers-settings
>>>>>>>>>>>>>> /var/ipfire/suricata/settings
>>>>>>>>>>>>>> /var/ipfire/suricata/suricata-default-rules.yaml
>>>>>>>>>>>>>> /var/ipfire/suricata/suricata-dns-servers.yaml
>>>>>>>>>>>>>> /var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
>>>>>>>>>>>>>> /var/ipfire/suricata/suricata-homenet.yaml
>>>>>>>>>>>>>> /var/ipfire/suricata/suricata-http-ports.yaml
>>>>>>>>>>>>>> /var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml
>>>>>>>>>>>>>> /var/ipfire/suricata/suricata-used-providers.yaml
>>>>>>>>>>>>>> /var/ipfire/time/
>>>>>>>>>>>>>> /var/ipfire/time/counter.conf
>>>>>>>>>>>>>> /var/ipfire/time/enable
>>>>>>>>>>>>>> /var/ipfire/time/settime.conf
>>>>>>>>>>>>>> /var/ipfire/time/settings
>>>>>>>>>>>>>> /var/ipfire/upnp/settings
>>>>>>>>>>>>>> /var/ipfire/urlfilter
>>>>>>>>>>>>>> /var/ipfire/urlfilter/settings
>>>>>>>>>>>>>> /var/ipfire/urlfilter/squidGuard.conf
>>>>>>>>>>>>>> /var/ipfire/vpn
>>>>>>>>>>>>>> /var/ipfire/vpn/config
>>>>>>>>>>>>>> /var/ipfire/vpn/ipsec.conf
>>>>>>>>>>>>>> /var/ipfire/vpn/settings
>>>>>>>>>>>>>> /var/ipfire/wakeonlan/clients.conf
>>>>>>>>>>>>>> /var/ipfire/wio/wio.conf
>>>>>>>>>>>>>> /var/ipfire/wireless/config
>>>>>>>>>>>>>> /var/ipfire/wireless/settings
>>>>>>>>>>>>>> /var/lib/suricata
>>>>>>>>>>>>>> /var/log/rrd/collectd
>>>>>>>>>>>>>> /var/log/rrd/hddshutdown-md127.rrd
>>>>>>>>>>>>>> /var/log/rrd/hddshutdown-sda.rrd
>>>>>>>>>>>>>> /var/log/rrd/hddshutdown-sdb.rrd
>>>>>>>>>>>>>> /var/log/rrd/hddtemp-md127.rrd
>>>>>>>>>>>>>> /var/log/rrd/hddtemp-sda.rrd
>>>>>>>>>>>>>> /var/log/rrd/hddtemp-sdb.rrd
>>>>>>>>>>>>>> /var/log/rrd/wio
>>>>>>>>>>>>>> /var/log/vnstat
>>>>>>>>>>>>>> /var/tmp/idsrules-emerging.tar.gz
>>>>>>>>>>>>>> /var/tmp/idsrules-sslbl_blacklist.rules
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Bernhard
>>>>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On 29 Mar 2022, at 14:10, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Just tried this patch out on my vm testbed system and it still doesn't work for me. The backup file had got to 1.3GB when I deleted the backup file as it was still growing. The normal correct backup file on that vm machine is around 7MB
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> The above was the case for both running it from the WUI or from the command line from my unprivileged user using sudo backupctrl exclude
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On 29/03/2022 14:27, Michael Tremer wrote:
>>>>>>>>>>>>>>>>>> This patch fixes globbing expansion in the backup include file list
>>>>>>>>>>>>>>>>>> which got broken in c7e0d73e7cfd7be95db9d0a5f3392b8241813d5b.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>> config/backup/backup.pl | 8 ++++----
>>>>>>>>>>>>>>>>>> 1 file changed, 4 insertions(+), 4 deletions(-)
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> diff --git a/config/backup/backup.pl b/config/backup/backup.pl
>>>>>>>>>>>>>>>>>> index a2337cf23..6f9295e94 100644
>>>>>>>>>>>>>>>>>> --- a/config/backup/backup.pl
>>>>>>>>>>>>>>>>>> +++ b/config/backup/backup.pl
>>>>>>>>>>>>>>>>>> @@ -19,6 +19,8 @@
>>>>>>>>>>>>>>>>>> # #
>>>>>>>>>>>>>>>>>> ###############################################################################
>>>>>>>>>>>>>>>>>> +shopt -s nullglob
>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>> NOW="$(date "+%Y-%m-%d-%H:%M")"
>>>>>>>>>>>>>>>>>> list_addons() {
>>>>>>>>>>>>>>>>>> @@ -38,10 +40,8 @@ process_includes() {
>>>>>>>>>>>>>>>>>> for include in $@; do
>>>>>>>>>>>>>>>>>> local file
>>>>>>>>>>>>>>>>>> while read -r file; do
>>>>>>>>>>>>>>>>>> -  for file in ${file}; do
>>>>>>>>>>>>>>>>>> -  if [ -e "/${file}" ]; then
>>>>>>>>>>>>>>>>>> -  echo "${file}"
>>>>>>>>>>>>>>>>>> -  fi
>>>>>>>>>>>>>>>>>> +  for file in /${file}; do
>>>>>>>>>>>>>>>>>> +  echo "${file}"
>>>>>>>>>>>>>>>>>> done
>>>>>>>>>>>>>>>>>> done < "${include}"
>>>>>>>>>>>>>>>>>> done | sort -u

Adolf Belka March 31, 2022, 6:50 p.m. UTC | #25

Hi Jon,

On 31/03/2022 20:22, Jon Murphy wrote:
> Adolf,
> 
> Might this issue be part of the overall issue?  I am wondering if it was 
> tested in the special release build?
> 
> https://bugzilla.ipfire.org/show_bug.cgi?id=12811 
> <https://bugzilla.ipfire.org/show_bug.cgi?id=12811>

I don't believe so. That bug already had a fix submitted into CU166 
before bug 12817 had a fix for the broken globbing expansion and I did a 
quick test when CU166 was still very early on and it seemed to work OK.

This current bug 12817 regarding globbing expansion is to do with not 
finding the files to be backed up.

Bug 12811 is to do with the restore process and not the backup. After 
the files have been restored from the backup the backup.pl routine was 
not able to find some of the misc programs used to restart various 
things in IPFire after the restore. The fix was a modified default patch 
to include the location where those misc progs are located.

Having said all the above, I could also be wrong. Maybe someone else can 
say if my analysis is right or flawed.

Regards,

Adolf.
> 
> Jon
> 
>> On Mar 31, 2022, at 5:16 AM, Adolf Belka <adolf.belka@ipfire.org 
>> <mailto:adolf.belka@ipfire.org>> wrote:
>>
>> Hi Michael,
>>
>> Update on the testing. I realised that I had done the test on a CU175 
>> version where I had modified the backup.pl code with pushd/popd.
>> I wouldn't expect that to give a problem but to confirm I created 
>> another CU165 clone with the backup.pl code in original form as most 
>> standard users would have it.
>> Re-ran the test and everything went correctly the same as my previous 
>> test.
>>
>> Just for security I also did a fresh install of CU166 from the latest 
>> master nightly.
>> That also worked fine. The backup had all the directories as expected.
>>
>> Regards,
>> Adolf
>>
>> On 31/03/2022 09:46, Michael Tremer wrote:
>>> Hello Adolf,
>>> Brilliant. Thank you very much for testing this in detail.
>>> Best,
>>> -Michael
>>>> On 31 Mar 2022, at 08:41, Adolf Belka <adolf.belka@ipfire.org 
>>>> <mailto:adolf.belka@ipfire.org>> wrote:
>>>>
>>>> Hi Michael,
>>>>
>>>> On 30/03/2022 23:20, Adolf Belka wrote:
>>>>> Hi,
>>>>>
>>>>> On 30/03/2022 17:05, Michael Tremer wrote:
>>>>>> Hello Bernhard,
>>>>>>
>>>>>> You are obviously right. My brain is absolute mush these days and 
>>>>>> doesn’t allow me to function.
>>>>>>
>>>>>> I will correct this by the end of the day.
>>>>>>
>>>>>> -Michael
>>>>>>
>>>>>>> On 30 Mar 2022, at 15:38, Bernhard Bitsch <bbitsch@ipfire.org 
>>>>>>> <mailto:bbitsch@ipfire.org>> wrote:
>>>>>>>
>>>>>>> Hello all,
>>>>>>>
>>>>>>>
>>>>>>> Am 30.03.2022 um 16:06 schrieb Michael Tremer:
>>>>>>>> Hello everyone,
>>>>>>>> Thank you very much for helping me test this.
>>>>>>>> I decided that we will have to create an emergency core update 
>>>>>>>> for this, because it will otherwise break people’s system. We 
>>>>>>>> also have a vulnerability in zlib which is being handed around 
>>>>>>>> by the press, so that has helped me to make a decision.
>>>>>>>
>>>>>>> I think this is a good solution. Shall we add a warning to backup 
>>>>>>> wiki page? Because in the moment a backup -> reinstall -> restore 
>>>>>>> process leaves an incomplete system.
>>>>>>>
>>>>>>>> I also pushed two more patches. One filters out any empty lines 
>>>>>>>> because they have quite a damaging effect and it happens easily 
>>>>>>>> that they are added to the include list. Secondly, I brought 
>>>>>>>> back the existence check which I thought was only there for the 
>>>>>>>> globbing mechanism, but also had the small side-effect of 
>>>>>>>> filtering out the file list early on which helps keeping tar happy.
>>>>>>>> Please review those changes and please install the new c166 from 
>>>>>>>> testing as soon as the build has finished (I am not sure if you 
>>>>>>>> are all on the nightly-builds list to see any notifications).
>>>>> I am not on the notifications list but I will check first thing 
>>>>> tomorrow and when the latest has been updated I will do a test 
>>>>> install and evaluation of the backup and provide confirmation of 
>>>>> how it functions.
>>>>>
>>>> As CU166 has been moved to master I took a clone of my CU165 vm and 
>>>> then did a CU on Testing with it rather than doing a fresh install. 
>>>> If you would also want a test of a fresh install done I can also do 
>>>> that.
>>>>
>>>> The version I tested was IPFire 2.27 (x86_64) - Core Update 166 
>>>> Development Build: master/8f696f60
>>>>
>>>> Anyway the backup is working fine again. All the directories are 
>>>> present. The size of the backup is now very similar to previously 
>>>> also confirming all contents should be good.
>>>>
>>>> I also added back in a couple of blank lines into my include.user 
>>>> file and the backup worked without any problems.
>>>>
>>>> I also did a restore of the backup and again as far as I can tell 
>>>> with a quick check through on my testbed, everything is fine.
>>>>
>>>> So everything looks good to me.
>>>>
>>>> Regards,
>>>>
>>>> Adolf.
>>>>
>>>>> Regards,
>>>>> Adolf.
>>>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=4f0e7f24f293ca10096e616010cfd4417afc157c 
>>>>>>>> <https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=4f0e7f24f293ca10096e616010cfd4417afc157c>
>>>>>>> The test on empty string should be in the 'while read -r file' 
>>>>>>> loop. The empty lines are in the include files. The outer for 
>>>>>>> loop iterates over the file list.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Bernhard
>>>>>>>
>>>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=b275771fdd10ac7116e5d400dc10d3148df8ac27 
>>>>>>>> <https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=b275771fdd10ac7116e5d400dc10d3148df8ac27>
>>>>>>>> Best,
>>>>>>>> -Michael
>>>>>>>>> On 29 Mar 2022, at 20:21, Bernhard Bitsch <bbitsch@ipfire.org> 
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Just checked the blank line problem.
>>>>>>>>> Pattern is empty , file name list is '/' --> no names produced 
>>>>>>>>> ( with existence check )
>>>>>>>>> Pattern is empty --> produced file name '/' ( without -e check )
>>>>>>>>>
>>>>>>>>> Am 29.03.2022 um 20:36 schrieb Bernhard Bitsch:
>>>>>>>>>> Hi all,
>>>>>>>>>> Am 29.03.2022 um 17:23 schrieb Adolf Belka:
>>>>>>>>>>> Hi All,
>>>>>>>>>>>
>>>>>>>>>>> On 29/03/2022 16:22, Michael Tremer wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> On 29 Mar 2022, at 15:22, Bernhard Bitsch 
>>>>>>>>>>>>> <bbitsch@ipfire.org> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Looks like your include.user file contains absolute paths.
>>>>>>>>>>> That was not the problem but I have corrected that. I would 
>>>>>>>>>>> have expected that when the main include and exclude files 
>>>>>>>>>>> were made relative that the update script would also have 
>>>>>>>>>>> checked for any entries in the include.user and exclude.user 
>>>>>>>>>>> files but it looks like it didn't.th
>>>>>>>>>> According to the source they are chosen.
>>>>>>>>>>>>
>>>>>>>>>>>> Or an empty line maybe?
>>>>>>>>>>> That turned out to be the problem. There was an empty line 
>>>>>>>>>>> after the two lines I had added. You only saw it if you 
>>>>>>>>>>> scrolled the pointer down the file in the editor.
>>>>>>>>>> Interesting. A blank line should produce nothing.
>>>>>>>>>>>
>>>>>>>>>>> Removing that blank line removed the line only with a / and 
>>>>>>>>>>> the backup then successfully ran and created a backup file 
>>>>>>>>>>> containing all the directories expected.
>>>>>>>>>>>
>>>>>>>>>>> The only thing I found is that there was the following tar 
>>>>>>>>>>> error message
>>>>>>>>>>>
>>>>>>>>>>> tar: Exiting with failure status due to previous errors
>>>>>>>>>>>
>>>>>>>>>>> The only thing I found in the backup output was several lines 
>>>>>>>>>>> such as
>>>>>>>>>>>
>>>>>>>>>>> tar: /etc/squid/squid.conf.local: Cannot stat: No such file 
>>>>>>>>>>> or directory
>>>>>>>>>>> tar: /etc/squid/squid.conf.pre.local: Cannot stat: No such 
>>>>>>>>>>> file or directory
>>>>>>>>>>>
>>>>>>>>>>> There were seven lines in total like this and the three I 
>>>>>>>>>>> checked were files that did not exist on my system. I presume 
>>>>>>>>>>> that these are files that should be backed up if present but 
>>>>>>>>>>> don't have to be present so that the error messages are 
>>>>>>>>>>> nothing to worry about.
>>>>>>>>>>> There were no error messages when doing the backup via the WUI.
>>>>>>>>>>>
>>>>>>>>>> Good find. ;)
>>>>>>>>>> Michael's patch removes the test of existence, globbing 
>>>>>>>>>> produces existent file names only. But I suppose there are 
>>>>>>>>>> several full names of files, not existing in each implementation.
>>>>>>>>>> Conclusion: Repair the file globbing by the shopt and do the 
>>>>>>>>>> existence check. So all existent files are chosen, that are 
>>>>>>>>>> listed in the include files.
>>>>>>>>>> Regards,
>>>>>>>>>> Bernhard
>>>>>>>>>>> The blank line was only on my vm testbed system but as I 
>>>>>>>>>>> clone this for any testing activities I will have to check 
>>>>>>>>>>> all of my existing vm's to fix this.
>>>>>>>>>>>
>>>>>>>>>>> Glad we resolved this simply. Sorry for the anxiety.
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>>
>>>>>>>>>>> Adolf.
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Am 29.03.2022 um 16:14 schrieb Adolf Belka:
>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>> This is what I get with the pushd/popd version in place on 
>>>>>>>>>>>>>> the same system
>>>>>>>>>>>>>> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
>>>>>>>>>>>>>> etc/group
>>>>>>>>>>>>>> etc/hosts
>>>>>>>>>>>>>> etc/hosts.allow
>>>>>>>>>>>>>> etc/hosts.deny
>>>>>>>>>>>>>> etc/httpd/server.crt
>>>>>>>>>>>>>> etc/httpd/server.csr
>>>>>>>>>>>>>> etc/httpd/server-ecdsa.crt
>>>>>>>>>>>>>> etc/httpd/server-ecdsa.csr
>>>>>>>>>>>>>> etc/httpd/server-ecdsa.key
>>>>>>>>>>>>>> etc/httpd/server.key
>>>>>>>>>>>>>> etc/ipsec.user.conf
>>>>>>>>>>>>>> etc/ipsec.user.secrets
>>>>>>>>>>>>>> etc/logrotate.d
>>>>>>>>>>>>>> etc/passwd
>>>>>>>>>>>>>> etc/shadow
>>>>>>>>>>>>>> etc/ssh/sshd_config
>>>>>>>>>>>>>> etc/ssh/ssh_host_ecdsa_key
>>>>>>>>>>>>>> etc/ssh/ssh_host_ecdsa_key.pub
>>>>>>>>>>>>>> etc/ssh/ssh_host_ed25519_key
>>>>>>>>>>>>>> etc/ssh/ssh_host_ed25519_key.pub
>>>>>>>>>>>>>> etc/ssh/ssh_host_rsa_key
>>>>>>>>>>>>>> etc/ssh/ssh_host_rsa_key.pub
>>>>>>>>>>>>>> /etc/sudoers
>>>>>>>>>>>>>> etc/sysconfig/createfiles
>>>>>>>>>>>>>> etc/sysconfig/firewall.local
>>>>>>>>>>>>>> etc/sysconfig/lm_sensors
>>>>>>>>>>>>>> etc/sysconfig/modules
>>>>>>>>>>>>>> etc/sysconfig/ramdisk
>>>>>>>>>>>>>> etc/sysconfig/rc
>>>>>>>>>>>>>> etc/sysconfig/rc.local
>>>>>>>>>>>>>> etc/unbound
>>>>>>>>>>>>>> /home/ahb
>>>>>>>>>>>>>> root/.bash_history
>>>>>>>>>>>>>> var/ipfire/accounting/settings.conf
>>>>>>>>>>>>>> var/ipfire/auth/users
>>>>>>>>>>>>>> var/ipfire/backup/addons/backup
>>>>>>>>>>>>>> var/ipfire/backup/exclude.user
>>>>>>>>>>>>>> var/ipfire/backup/include.user
>>>>>>>>>>>>>> var/ipfire/ca/cacert.pem
>>>>>>>>>>>>>> var/ipfire/captive/agb.txt
>>>>>>>>>>>>>> var/ipfire/captive/clients
>>>>>>>>>>>>>> var/ipfire/captive/coupons
>>>>>>>>>>>>>> var/ipfire/captive/logo.dat
>>>>>>>>>>>>>> var/ipfire/captive/settings
>>>>>>>>>>>>>> var/ipfire/captive/terms.txt
>>>>>>>>>>>>>> var/ipfire/captive/voucher_out
>>>>>>>>>>>>>> var/ipfire/certs/hostcert.pem
>>>>>>>>>>>>>> var/ipfire/certs/hostkey.pem
>>>>>>>>>>>>>> var/ipfire/certs/phoebevmipseccert.pem
>>>>>>>>>>>>>> var/ipfire/connscheduler/connscheduler.conf
>>>>>>>>>>>>>> var/ipfire/crls/cacrl.pem
>>>>>>>>>>>>>> var/ipfire/cups/cups-browsed.conf
>>>>>>>>>>>>>> var/ipfire/cups/subscriptions.conf
>>>>>>>>>>>>>> var/ipfire/ddns/config
>>>>>>>>>>>>>> var/ipfire/ddns/ddns.conf
>>>>>>>>>>>>>> var/ipfire/ddns/settings
>>>>>>>>>>>>>> var/ipfire/dhcp/advoptions
>>>>>>>>>>>>>> var/ipfire/dhcp/advoptions-list
>>>>>>>>>>>>>> var/ipfire/dhcpc/dhcpcd.conf
>>>>>>>>>>>>>> var/ipfire/dhcp/dhcpd.conf
>>>>>>>>>>>>>> var/ipfire/dhcp/dhcpd.conf.local
>>>>>>>>>>>>>> var/ipfire/dhcp/enable_blue
>>>>>>>>>>>>>> var/ipfire/dhcp/enable_green
>>>>>>>>>>>>>> var/ipfire/dhcp/fixleases
>>>>>>>>>>>>>> var/ipfire/dhcp/settings
>>>>>>>>>>>>>> var/ipfire/dma/auth.conf
>>>>>>>>>>>>>> var/ipfire/dma/dma.conf
>>>>>>>>>>>>>> var/ipfire/dma/mail.conf
>>>>>>>>>>>>>> var/ipfire/dns
>>>>>>>>>>>>>> var/ipfire/dnsforward/config
>>>>>>>>>>>>>> var/ipfire/dns/settings
>>>>>>>>>>>>>> var/ipfire/ethernet/aliases
>>>>>>>>>>>>>> var/ipfire/ethernet/settings
>>>>>>>>>>>>>> var/ipfire/ethernet/wireless
>>>>>>>>>>>>>> var/ipfire/extrahd/settings
>>>>>>>>>>>>>> var/ipfire/firewall
>>>>>>>>>>>>>> var/ipfire/firewall/config
>>>>>>>>>>>>>> var/ipfire/firewall/settings
>>>>>>>>>>>>>> var/ipfire/fwhosts
>>>>>>>>>>>>>> var/ipfire/isdn/settings
>>>>>>>>>>>>>> var/ipfire/logging/settings
>>>>>>>>>>>>>> var/ipfire/mac/settings
>>>>>>>>>>>>>> var/ipfire/main/firstsetup_ok
>>>>>>>>>>>>>> var/ipfire/main/gpl_accepted
>>>>>>>>>>>>>> var/ipfire/main/hostname.conf
>>>>>>>>>>>>>> var/ipfire/main/hosts
>>>>>>>>>>>>>> var/ipfire/main/manualpages
>>>>>>>>>>>>>> var/ipfire/main/routing
>>>>>>>>>>>>>> var/ipfire/main/security
>>>>>>>>>>>>>> var/ipfire/main/send_profile
>>>>>>>>>>>>>> var/ipfire/main/settings
>>>>>>>>>>>>>> var/ipfire/modem/settings
>>>>>>>>>>>>>> var/ipfire/optionsfw/settings
>>>>>>>>>>>>>> var/ipfire/ovpn
>>>>>>>>>>>>>> var/ipfire/ovpn/ccd.conf
>>>>>>>>>>>>>> var/ipfire/ovpn/collectd.vpn
>>>>>>>>>>>>>> var/ipfire/ovpn/enable
>>>>>>>>>>>>>> var/ipfire/ovpn/server.conf
>>>>>>>>>>>>>> var/ipfire/ovpn/settings
>>>>>>>>>>>>>> var/ipfire/pakfire/settings
>>>>>>>>>>>>>> var/ipfire/ppp
>>>>>>>>>>>>>> var/ipfire/ppp/fake-resolv.conf
>>>>>>>>>>>>>> var/ipfire/ppp/settings
>>>>>>>>>>>>>> var/ipfire/private/cakey.pem
>>>>>>>>>>>>>> var/ipfire/proxy
>>>>>>>>>>>>>> var/ipfire/proxy/asnbl-helper.conf
>>>>>>>>>>>>>> var/ipfire/proxy/cachemgr.conf
>>>>>>>>>>>>>> var/ipfire/proxy/enable
>>>>>>>>>>>>>> var/ipfire/proxy/settings
>>>>>>>>>>>>>> var/ipfire/proxy/squid.conf
>>>>>>>>>>>>>> var/ipfire/qos/bin
>>>>>>>>>>>>>> var/ipfire/qos/classes
>>>>>>>>>>>>>> var/ipfire/qos/level7config
>>>>>>>>>>>>>> var/ipfire/qos/portconfig
>>>>>>>>>>>>>> var/ipfire/qos/settings
>>>>>>>>>>>>>> var/ipfire/qos/subclasses
>>>>>>>>>>>>>> var/ipfire/qos/tosconfig
>>>>>>>>>>>>>> var/ipfire/remote/enablessh
>>>>>>>>>>>>>> var/ipfire/remote/settings
>>>>>>>>>>>>>> var/ipfire/sensors/settings
>>>>>>>>>>>>>> var/ipfire/suricata/oinkmaster.conf
>>>>>>>>>>>>>> var/ipfire/suricata/oinkmaster-modify-sids.conf
>>>>>>>>>>>>>> var/ipfire/suricata/oinkmaster-provider-includes.conf
>>>>>>>>>>>>>> var/ipfire/suricata/providers-settings
>>>>>>>>>>>>>> var/ipfire/suricata/settings
>>>>>>>>>>>>>> var/ipfire/suricata/suricata-default-rules.yaml
>>>>>>>>>>>>>> var/ipfire/suricata/suricata-dns-servers.yaml
>>>>>>>>>>>>>> var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
>>>>>>>>>>>>>> var/ipfire/suricata/suricata-homenet.yaml
>>>>>>>>>>>>>> var/ipfire/suricata/suricata-http-ports.yaml
>>>>>>>>>>>>>> var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml
>>>>>>>>>>>>>> var/ipfire/suricata/suricata-used-providers.yaml
>>>>>>>>>>>>>> var/ipfire/time/
>>>>>>>>>>>>>> var/ipfire/time/counter.conf
>>>>>>>>>>>>>> var/ipfire/time/enable
>>>>>>>>>>>>>> var/ipfire/time/settime.conf
>>>>>>>>>>>>>> var/ipfire/time/settings
>>>>>>>>>>>>>> var/ipfire/upnp/settings
>>>>>>>>>>>>>> var/ipfire/urlfilter
>>>>>>>>>>>>>> var/ipfire/urlfilter/settings
>>>>>>>>>>>>>> var/ipfire/urlfilter/squidGuard.conf
>>>>>>>>>>>>>> var/ipfire/vpn
>>>>>>>>>>>>>> var/ipfire/vpn/config
>>>>>>>>>>>>>> var/ipfire/vpn/ipsec.conf
>>>>>>>>>>>>>> var/ipfire/vpn/settings
>>>>>>>>>>>>>> var/ipfire/wakeonlan/clients.conf
>>>>>>>>>>>>>> var/ipfire/wio/wio.conf
>>>>>>>>>>>>>> var/ipfire/wireless/config
>>>>>>>>>>>>>> var/ipfire/wireless/settings
>>>>>>>>>>>>>> var/lib/suricata
>>>>>>>>>>>>>> var/log/rrd/collectd
>>>>>>>>>>>>>> var/log/rrd/hddshutdown-md127.rrd
>>>>>>>>>>>>>> var/log/rrd/hddshutdown-sda.rrd
>>>>>>>>>>>>>> var/log/rrd/hddshutdown-sdb.rrd
>>>>>>>>>>>>>> var/log/rrd/hddtemp-md127.rrd
>>>>>>>>>>>>>> var/log/rrd/hddtemp-sda.rrd
>>>>>>>>>>>>>> var/log/rrd/hddtemp-sdb.rrd
>>>>>>>>>>>>>> var/log/rrd/wio
>>>>>>>>>>>>>> var/log/vnstat
>>>>>>>>>>>>>> var/tmp/idsrules-emerging.tar.gz
>>>>>>>>>>>>>> var/tmp/idsrules-sslbl_blacklist.rules
>>>>>>>>>>>>>> The following are in the previous list but not in this one:-
>>>>>>>>>>>>>> /
>>>>>>>>>>>>>> /etc/conntrackd/conntrackd.conf
>>>>>>>>>>>>>> /etc/ipsec.user-post.conf
>>>>>>>>>>>>>> /root/.gitconfig
>>>>>>>>>>>>>> /root/.ssh
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>> On 29/03/2022 16:01, Adolf Belka wrote:
>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 29/03/2022 15:36, Bernhard Bitsch wrote:
>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Am 29.03.2022 um 15:11 schrieb Michael Tremer:
>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Could you please send the file listing to find out what 
>>>>>>>>>>>>>>>>> is being included what shouldn’t?
>>>>>>>>>>>>>>> Unfortunately, as I stopped the backup continuing once it 
>>>>>>>>>>>>>>> had reached 1.2GB, the file created was not able to be 
>>>>>>>>>>>>>>> opened. Probably stopping the backup corrupted it in some 
>>>>>>>>>>>>>>> way.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Just for easy quick handling, you can use
>>>>>>>>>>>>>>>> '/var/ipfire/backup/bin/backup.pl list' to show the 
>>>>>>>>>>>>>>>> files included.
>>>>>>>>>>>>>>> I gave this a go and here is the output from the command.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
>>>>>>>>>>>>>>> /
>>>>>>>>>>>>>>> /etc/conntrackd/conntrackd.conf
>>>>>>>>>>>>>>> /etc/group
>>>>>>>>>>>>>>> /etc/hosts
>>>>>>>>>>>>>>> /etc/hosts.allow
>>>>>>>>>>>>>>> /etc/hosts.deny
>>>>>>>>>>>>>>> /etc/httpd/server.crt
>>>>>>>>>>>>>>> /etc/httpd/server.csr
>>>>>>>>>>>>>>> /etc/httpd/server-ecdsa.crt
>>>>>>>>>>>>>>> /etc/httpd/server-ecdsa.csr
>>>>>>>>>>>>>>> /etc/httpd/server-ecdsa.key
>>>>>>>>>>>>>>> /etc/httpd/server.key
>>>>>>>>>>>>>>> /etc/ipsec.user.conf
>>>>>>>>>>>>>>> /etc/ipsec.user-post.conf
>>>>>>>>>>>>>>> /etc/ipsec.user.secrets
>>>>>>>>>>>>>>> /etc/logrotate.d
>>>>>>>>>>>>>>> /etc/passwd
>>>>>>>>>>>>>>> /etc/shadow
>>>>>>>>>>>>>>> /etc/squid/squid.conf.local
>>>>>>>>>>>>>>> /etc/squid/squid.conf.pre.local
>>>>>>>>>>>>>>> /etc/ssh/sshd_config
>>>>>>>>>>>>>>> /etc/ssh/ssh_host_ecdsa_key
>>>>>>>>>>>>>>> /etc/ssh/ssh_host_ecdsa_key.pub
>>>>>>>>>>>>>>> /etc/ssh/ssh_host_ed25519_key
>>>>>>>>>>>>>>> /etc/ssh/ssh_host_ed25519_key.pub
>>>>>>>>>>>>>>> /etc/ssh/ssh_host_rsa_key
>>>>>>>>>>>>>>> /etc/ssh/ssh_host_rsa_key.pub
>>>>>>>>>>>>>>> //etc/sudoers
>>>>>>>>>>>>>>> /etc/sysconfig/createfiles
>>>>>>>>>>>>>>> /etc/sysconfig/firewall.local
>>>>>>>>>>>>>>> /etc/sysconfig/lm_sensors
>>>>>>>>>>>>>>> /etc/sysconfig/modules
>>>>>>>>>>>>>>> /etc/sysconfig/ramdisk
>>>>>>>>>>>>>>> /etc/sysconfig/rc
>>>>>>>>>>>>>>> /etc/sysconfig/rc.local
>>>>>>>>>>>>>>> /etc/unbound
>>>>>>>>>>>>>>> //home/ahb
>>>>>>>>>>>>>>> /root/.bash_history
>>>>>>>>>>>>>>> /root/.gitconfig
>>>>>>>>>>>>>>> /root/.ssh
>>>>>>>>>>>>>>> /var/ipfire/accounting/settings.conf
>>>>>>>>>>>>>>> /var/ipfire/auth/users
>>>>>>>>>>>>>>> /var/ipfire/backup/addons/backup
>>>>>>>>>>>>>>> /var/ipfire/backup/exclude.user
>>>>>>>>>>>>>>> /var/ipfire/backup/include.user
>>>>>>>>>>>>>>> /var/ipfire/ca/cacert.pem
>>>>>>>>>>>>>>> /var/ipfire/captive/agb.txt
>>>>>>>>>>>>>>> /var/ipfire/captive/clients
>>>>>>>>>>>>>>> /var/ipfire/captive/coupons
>>>>>>>>>>>>>>> /var/ipfire/captive/logo.dat
>>>>>>>>>>>>>>> /var/ipfire/captive/settings
>>>>>>>>>>>>>>> /var/ipfire/captive/terms.txt
>>>>>>>>>>>>>>> /var/ipfire/captive/voucher_out
>>>>>>>>>>>>>>> /var/ipfire/certs/hostcert.pem
>>>>>>>>>>>>>>> /var/ipfire/certs/hostkey.pem
>>>>>>>>>>>>>>> /var/ipfire/certs/phoebevmipseccert.pem
>>>>>>>>>>>>>>> /var/ipfire/connscheduler/connscheduler.conf
>>>>>>>>>>>>>>> /var/ipfire/crls/cacrl.pem
>>>>>>>>>>>>>>> /var/ipfire/cups/cups-browsed.conf
>>>>>>>>>>>>>>> /var/ipfire/cups/subscriptions.conf
>>>>>>>>>>>>>>> /var/ipfire/ddns/config
>>>>>>>>>>>>>>> /var/ipfire/ddns/ddns.conf
>>>>>>>>>>>>>>> /var/ipfire/ddns/settings
>>>>>>>>>>>>>>> /var/ipfire/dhcp/advoptions
>>>>>>>>>>>>>>> /var/ipfire/dhcp/advoptions-list
>>>>>>>>>>>>>>> /var/ipfire/dhcpc/dhcpcd.conf
>>>>>>>>>>>>>>> /var/ipfire/dhcp/dhcpd.conf
>>>>>>>>>>>>>>> /var/ipfire/dhcp/dhcpd.conf.local
>>>>>>>>>>>>>>> /var/ipfire/dhcp/enable_blue
>>>>>>>>>>>>>>> /var/ipfire/dhcp/enable_green
>>>>>>>>>>>>>>> /var/ipfire/dhcp/fixleases
>>>>>>>>>>>>>>> /var/ipfire/dhcp/settings
>>>>>>>>>>>>>>> /var/ipfire/dma/auth.conf
>>>>>>>>>>>>>>> /var/ipfire/dma/dma.conf
>>>>>>>>>>>>>>> /var/ipfire/dma/mail.conf
>>>>>>>>>>>>>>> /var/ipfire/dns
>>>>>>>>>>>>>>> /var/ipfire/dnsforward/config
>>>>>>>>>>>>>>> /var/ipfire/dns/settings
>>>>>>>>>>>>>>> /var/ipfire/ethernet/aliases
>>>>>>>>>>>>>>> /var/ipfire/ethernet/settings
>>>>>>>>>>>>>>> /var/ipfire/ethernet/wireless
>>>>>>>>>>>>>>> /var/ipfire/extrahd/settings
>>>>>>>>>>>>>>> /var/ipfire/firewall
>>>>>>>>>>>>>>> /var/ipfire/firewall/config
>>>>>>>>>>>>>>> /var/ipfire/firewall/settings
>>>>>>>>>>>>>>> /var/ipfire/fwhosts
>>>>>>>>>>>>>>> /var/ipfire/isdn/settings
>>>>>>>>>>>>>>> /var/ipfire/logging/settings
>>>>>>>>>>>>>>> /var/ipfire/mac/settings
>>>>>>>>>>>>>>> /var/ipfire/main/firstsetup_ok
>>>>>>>>>>>>>>> /var/ipfire/main/gpl_accepted
>>>>>>>>>>>>>>> /var/ipfire/main/hostname.conf
>>>>>>>>>>>>>>> /var/ipfire/main/hosts
>>>>>>>>>>>>>>> /var/ipfire/main/manualpages
>>>>>>>>>>>>>>> /var/ipfire/main/routing
>>>>>>>>>>>>>>> /var/ipfire/main/security
>>>>>>>>>>>>>>> /var/ipfire/main/send_profile
>>>>>>>>>>>>>>> /var/ipfire/main/settings
>>>>>>>>>>>>>>> /var/ipfire/modem/settings
>>>>>>>>>>>>>>> /var/ipfire/optionsfw/settings
>>>>>>>>>>>>>>> /var/ipfire/ovpn
>>>>>>>>>>>>>>> /var/ipfire/ovpn/ccd.conf
>>>>>>>>>>>>>>> /var/ipfire/ovpn/collectd.vpn
>>>>>>>>>>>>>>> /var/ipfire/ovpn/enable
>>>>>>>>>>>>>>> /var/ipfire/ovpn/server.conf
>>>>>>>>>>>>>>> /var/ipfire/ovpn/settings
>>>>>>>>>>>>>>> /var/ipfire/pakfire/settings
>>>>>>>>>>>>>>> /var/ipfire/ppp
>>>>>>>>>>>>>>> /var/ipfire/ppp/fake-resolv.conf
>>>>>>>>>>>>>>> /var/ipfire/ppp/settings
>>>>>>>>>>>>>>> /var/ipfire/private/cakey.pem
>>>>>>>>>>>>>>> /var/ipfire/proxy
>>>>>>>>>>>>>>> /var/ipfire/proxy/asnbl-helper.conf
>>>>>>>>>>>>>>> /var/ipfire/proxy/cachemgr.conf
>>>>>>>>>>>>>>> /var/ipfire/proxy/enable
>>>>>>>>>>>>>>> /var/ipfire/proxy/settings
>>>>>>>>>>>>>>> /var/ipfire/proxy/squid.conf
>>>>>>>>>>>>>>> /var/ipfire/qos/bin
>>>>>>>>>>>>>>> /var/ipfire/qos/bin/qos.sh
>>>>>>>>>>>>>>> /var/ipfire/qos/classes
>>>>>>>>>>>>>>> /var/ipfire/qos/level7config
>>>>>>>>>>>>>>> /var/ipfire/qos/portconfig
>>>>>>>>>>>>>>> /var/ipfire/qos/settings
>>>>>>>>>>>>>>> /var/ipfire/qos/subclasses
>>>>>>>>>>>>>>> /var/ipfire/qos/tosconfig
>>>>>>>>>>>>>>> /var/ipfire/remote/enablessh
>>>>>>>>>>>>>>> /var/ipfire/remote/settings
>>>>>>>>>>>>>>> /var/ipfire/sensors/settings
>>>>>>>>>>>>>>> /var/ipfire/suricata/oinkmaster.conf
>>>>>>>>>>>>>>> /var/ipfire/suricata/oinkmaster-modify-sids.conf
>>>>>>>>>>>>>>> /var/ipfire/suricata/oinkmaster-provider-includes.conf
>>>>>>>>>>>>>>> /var/ipfire/suricata/providers-settings
>>>>>>>>>>>>>>> /var/ipfire/suricata/settings
>>>>>>>>>>>>>>> /var/ipfire/suricata/suricata-default-rules.yaml
>>>>>>>>>>>>>>> /var/ipfire/suricata/suricata-dns-servers.yaml
>>>>>>>>>>>>>>> /var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
>>>>>>>>>>>>>>> /var/ipfire/suricata/suricata-homenet.yaml
>>>>>>>>>>>>>>> /var/ipfire/suricata/suricata-http-ports.yaml
>>>>>>>>>>>>>>> /var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml
>>>>>>>>>>>>>>> /var/ipfire/suricata/suricata-used-providers.yaml
>>>>>>>>>>>>>>> /var/ipfire/time/
>>>>>>>>>>>>>>> /var/ipfire/time/counter.conf
>>>>>>>>>>>>>>> /var/ipfire/time/enable
>>>>>>>>>>>>>>> /var/ipfire/time/settime.conf
>>>>>>>>>>>>>>> /var/ipfire/time/settings
>>>>>>>>>>>>>>> /var/ipfire/upnp/settings
>>>>>>>>>>>>>>> /var/ipfire/urlfilter
>>>>>>>>>>>>>>> /var/ipfire/urlfilter/settings
>>>>>>>>>>>>>>> /var/ipfire/urlfilter/squidGuard.conf
>>>>>>>>>>>>>>> /var/ipfire/vpn
>>>>>>>>>>>>>>> /var/ipfire/vpn/config
>>>>>>>>>>>>>>> /var/ipfire/vpn/ipsec.conf
>>>>>>>>>>>>>>> /var/ipfire/vpn/settings
>>>>>>>>>>>>>>> /var/ipfire/wakeonlan/clients.conf
>>>>>>>>>>>>>>> /var/ipfire/wio/wio.conf
>>>>>>>>>>>>>>> /var/ipfire/wireless/config
>>>>>>>>>>>>>>> /var/ipfire/wireless/settings
>>>>>>>>>>>>>>> /var/lib/suricata
>>>>>>>>>>>>>>> /var/log/rrd/collectd
>>>>>>>>>>>>>>> /var/log/rrd/hddshutdown-md127.rrd
>>>>>>>>>>>>>>> /var/log/rrd/hddshutdown-sda.rrd
>>>>>>>>>>>>>>> /var/log/rrd/hddshutdown-sdb.rrd
>>>>>>>>>>>>>>> /var/log/rrd/hddtemp-md127.rrd
>>>>>>>>>>>>>>> /var/log/rrd/hddtemp-sda.rrd
>>>>>>>>>>>>>>> /var/log/rrd/hddtemp-sdb.rrd
>>>>>>>>>>>>>>> /var/log/rrd/wio
>>>>>>>>>>>>>>> /var/log/vnstat
>>>>>>>>>>>>>>> /var/tmp/idsrules-emerging.tar.gz
>>>>>>>>>>>>>>> /var/tmp/idsrules-sslbl_blacklist.rules
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Bernhard
>>>>>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On 29 Mar 2022, at 14:10, Adolf Belka 
>>>>>>>>>>>>>>>>>> <adolf.belka@ipfire.org> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Just tried this patch out on my vm testbed system and 
>>>>>>>>>>>>>>>>>> it still doesn't work for me. The backup file had got 
>>>>>>>>>>>>>>>>>> to 1.3GB when I deleted the backup file as it was 
>>>>>>>>>>>>>>>>>> still growing. The normal correct backup file on that 
>>>>>>>>>>>>>>>>>> vm machine is around 7MB
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The above was the case for both running it from the 
>>>>>>>>>>>>>>>>>> WUI or from the command line from my unprivileged user 
>>>>>>>>>>>>>>>>>> using sudo backupctrl exclude
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On 29/03/2022 14:27, Michael Tremer wrote:
>>>>>>>>>>>>>>>>>>> This patch fixes globbing expansion in the backup 
>>>>>>>>>>>>>>>>>>> include file list
>>>>>>>>>>>>>>>>>>> which got broken in 
>>>>>>>>>>>>>>>>>>> c7e0d73e7cfd7be95db9d0a5f3392b8241813d5b.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
>>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>> config/backup/backup.pl | 8 ++++----
>>>>>>>>>>>>>>>>>>> 1 file changed, 4 insertions(+), 4 deletions(-)
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> diff --git a/config/backup/backup.pl 
>>>>>>>>>>>>>>>>>>> b/config/backup/backup.pl
>>>>>>>>>>>>>>>>>>> index a2337cf23..6f9295e94 100644
>>>>>>>>>>>>>>>>>>> --- a/config/backup/backup.pl
>>>>>>>>>>>>>>>>>>> +++ b/config/backup/backup.pl
>>>>>>>>>>>>>>>>>>> @@ -19,6 +19,8 @@
>>>>>>>>>>>>>>>>>>> # #
>>>>>>>>>>>>>>>>>>> ###############################################################################
>>>>>>>>>>>>>>>>>>> +shopt -s nullglob
>>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>>> NOW="$(date "+%Y-%m-%d-%H:%M")"
>>>>>>>>>>>>>>>>>>> list_addons() {
>>>>>>>>>>>>>>>>>>> @@ -38,10 +40,8 @@ process_includes() {
>>>>>>>>>>>>>>>>>>> for include in $@; do
>>>>>>>>>>>>>>>>>>> local file
>>>>>>>>>>>>>>>>>>> while read -r file; do
>>>>>>>>>>>>>>>>>>> -  for file in ${file}; do
>>>>>>>>>>>>>>>>>>> -  if [ -e "/${file}" ]; then
>>>>>>>>>>>>>>>>>>> -  echo "${file}"
>>>>>>>>>>>>>>>>>>> -  fi
>>>>>>>>>>>>>>>>>>> +  for file in /${file}; do
>>>>>>>>>>>>>>>>>>> +  echo "${file}"
>>>>>>>>>>>>>>>>>>> done
>>>>>>>>>>>>>>>>>>> done < "${include}"
>>>>>>>>>>>>>>>>>>> done | sort -u
>

Bernhard Bitsch March 31, 2022, 7:17 p.m. UTC | #26

Hi,

I don't think so, also.

Bug #12811 shows a problem with the PATH variable ( for new installations ).
The progs mentioned in bugzilla case should be callable.

Bug #12817 is about missing files in the backup. As Adolf stated, the 
programs should be found independent from the existence of the data 
files from the backup.

Regards,
Bernhard

Am 31.03.2022 um 20:50 schrieb Adolf Belka:
> Hi Jon,
> 
> On 31/03/2022 20:22, Jon Murphy wrote:
>> Adolf,
>>
>> Might this issue be part of the overall issue?  I am wondering if it 
>> was tested in the special release build?
>>
>> https://bugzilla.ipfire.org/show_bug.cgi?id=12811 
>> <https://bugzilla.ipfire.org/show_bug.cgi?id=12811>
> 
> I don't believe so. That bug already had a fix submitted into CU166 
> before bug 12817 had a fix for the broken globbing expansion and I did a 
> quick test when CU166 was still very early on and it seemed to work OK.
> 
> This current bug 12817 regarding globbing expansion is to do with not 
> finding the files to be backed up.
> 
> Bug 12811 is to do with the restore process and not the backup. After 
> the files have been restored from the backup the backup.pl routine was 
> not able to find some of the misc programs used to restart various 
> things in IPFire after the restore. The fix was a modified default patch 
> to include the location where those misc progs are located.
> 
> Having said all the above, I could also be wrong. Maybe someone else can 
> say if my analysis is right or flawed.
> 
> Regards,
> 
> Adolf.
>>
>> Jon
>>
>>> On Mar 31, 2022, at 5:16 AM, Adolf Belka <adolf.belka@ipfire.org 
>>> <mailto:adolf.belka@ipfire.org>> wrote:
>>>
>>> Hi Michael,
>>>
>>> Update on the testing. I realised that I had done the test on a CU175 
>>> version where I had modified the backup.pl code with pushd/popd.
>>> I wouldn't expect that to give a problem but to confirm I created 
>>> another CU165 clone with the backup.pl code in original form as most 
>>> standard users would have it.
>>> Re-ran the test and everything went correctly the same as my previous 
>>> test.
>>>
>>> Just for security I also did a fresh install of CU166 from the latest 
>>> master nightly.
>>> That also worked fine. The backup had all the directories as expected.
>>>
>>> Regards,
>>> Adolf
>>>
>>> On 31/03/2022 09:46, Michael Tremer wrote:
>>>> Hello Adolf,
>>>> Brilliant. Thank you very much for testing this in detail.
>>>> Best,
>>>> -Michael
>>>>> On 31 Mar 2022, at 08:41, Adolf Belka <adolf.belka@ipfire.org 
>>>>> <mailto:adolf.belka@ipfire.org>> wrote:
>>>>>
>>>>> Hi Michael,
>>>>>
>>>>> On 30/03/2022 23:20, Adolf Belka wrote:
>>>>>> Hi,
>>>>>>
>>>>>> On 30/03/2022 17:05, Michael Tremer wrote:
>>>>>>> Hello Bernhard,
>>>>>>>
>>>>>>> You are obviously right. My brain is absolute mush these days and 
>>>>>>> doesn’t allow me to function.
>>>>>>>
>>>>>>> I will correct this by the end of the day.
>>>>>>>
>>>>>>> -Michael
>>>>>>>
>>>>>>>> On 30 Mar 2022, at 15:38, Bernhard Bitsch <bbitsch@ipfire.org 
>>>>>>>> <mailto:bbitsch@ipfire.org>> wrote:
>>>>>>>>
>>>>>>>> Hello all,
>>>>>>>>
>>>>>>>>
>>>>>>>> Am 30.03.2022 um 16:06 schrieb Michael Tremer:
>>>>>>>>> Hello everyone,
>>>>>>>>> Thank you very much for helping me test this.
>>>>>>>>> I decided that we will have to create an emergency core update 
>>>>>>>>> for this, because it will otherwise break people’s system. We 
>>>>>>>>> also have a vulnerability in zlib which is being handed around 
>>>>>>>>> by the press, so that has helped me to make a decision.
>>>>>>>>
>>>>>>>> I think this is a good solution. Shall we add a warning to 
>>>>>>>> backup wiki page? Because in the moment a backup -> reinstall -> 
>>>>>>>> restore process leaves an incomplete system.
>>>>>>>>
>>>>>>>>> I also pushed two more patches. One filters out any empty lines 
>>>>>>>>> because they have quite a damaging effect and it happens easily 
>>>>>>>>> that they are added to the include list. Secondly, I brought 
>>>>>>>>> back the existence check which I thought was only there for the 
>>>>>>>>> globbing mechanism, but also had the small side-effect of 
>>>>>>>>> filtering out the file list early on which helps keeping tar 
>>>>>>>>> happy.
>>>>>>>>> Please review those changes and please install the new c166 
>>>>>>>>> from testing as soon as the build has finished (I am not sure 
>>>>>>>>> if you are all on the nightly-builds list to see any 
>>>>>>>>> notifications).
>>>>>> I am not on the notifications list but I will check first thing 
>>>>>> tomorrow and when the latest has been updated I will do a test 
>>>>>> install and evaluation of the backup and provide confirmation of 
>>>>>> how it functions.
>>>>>>
>>>>> As CU166 has been moved to master I took a clone of my CU165 vm and 
>>>>> then did a CU on Testing with it rather than doing a fresh install. 
>>>>> If you would also want a test of a fresh install done I can also do 
>>>>> that.
>>>>>
>>>>> The version I tested was IPFire 2.27 (x86_64) - Core Update 166 
>>>>> Development Build: master/8f696f60
>>>>>
>>>>> Anyway the backup is working fine again. All the directories are 
>>>>> present. The size of the backup is now very similar to previously 
>>>>> also confirming all contents should be good.
>>>>>
>>>>> I also added back in a couple of blank lines into my include.user 
>>>>> file and the backup worked without any problems.
>>>>>
>>>>> I also did a restore of the backup and again as far as I can tell 
>>>>> with a quick check through on my testbed, everything is fine.
>>>>>
>>>>> So everything looks good to me.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Adolf.
>>>>>
>>>>>> Regards,
>>>>>> Adolf.
>>>>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=4f0e7f24f293ca10096e616010cfd4417afc157c 
>>>>>>>>> <https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=4f0e7f24f293ca10096e616010cfd4417afc157c> 
>>>>>>>>>
>>>>>>>> The test on empty string should be in the 'while read -r file' 
>>>>>>>> loop. The empty lines are in the include files. The outer for 
>>>>>>>> loop iterates over the file list.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Bernhard
>>>>>>>>
>>>>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=b275771fdd10ac7116e5d400dc10d3148df8ac27 
>>>>>>>>> <https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=b275771fdd10ac7116e5d400dc10d3148df8ac27> 
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> -Michael
>>>>>>>>>> On 29 Mar 2022, at 20:21, Bernhard Bitsch <bbitsch@ipfire.org> 
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Just checked the blank line problem.
>>>>>>>>>> Pattern is empty , file name list is '/' --> no names produced 
>>>>>>>>>> ( with existence check )
>>>>>>>>>> Pattern is empty --> produced file name '/' ( without -e check )
>>>>>>>>>>
>>>>>>>>>> Am 29.03.2022 um 20:36 schrieb Bernhard Bitsch:
>>>>>>>>>>> Hi all,
>>>>>>>>>>> Am 29.03.2022 um 17:23 schrieb Adolf Belka:
>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>
>>>>>>>>>>>> On 29/03/2022 16:22, Michael Tremer wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 29 Mar 2022, at 15:22, Bernhard Bitsch 
>>>>>>>>>>>>>> <bbitsch@ipfire.org> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Looks like your include.user file contains absolute paths.
>>>>>>>>>>>> That was not the problem but I have corrected that. I would 
>>>>>>>>>>>> have expected that when the main include and exclude files 
>>>>>>>>>>>> were made relative that the update script would also have 
>>>>>>>>>>>> checked for any entries in the include.user and exclude.user 
>>>>>>>>>>>> files but it looks like it didn't.th
>>>>>>>>>>> According to the source they are chosen.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Or an empty line maybe?
>>>>>>>>>>>> That turned out to be the problem. There was an empty line 
>>>>>>>>>>>> after the two lines I had added. You only saw it if you 
>>>>>>>>>>>> scrolled the pointer down the file in the editor.
>>>>>>>>>>> Interesting. A blank line should produce nothing.
>>>>>>>>>>>>
>>>>>>>>>>>> Removing that blank line removed the line only with a / and 
>>>>>>>>>>>> the backup then successfully ran and created a backup file 
>>>>>>>>>>>> containing all the directories expected.
>>>>>>>>>>>>
>>>>>>>>>>>> The only thing I found is that there was the following tar 
>>>>>>>>>>>> error message
>>>>>>>>>>>>
>>>>>>>>>>>> tar: Exiting with failure status due to previous errors
>>>>>>>>>>>>
>>>>>>>>>>>> The only thing I found in the backup output was several 
>>>>>>>>>>>> lines such as
>>>>>>>>>>>>
>>>>>>>>>>>> tar: /etc/squid/squid.conf.local: Cannot stat: No such file 
>>>>>>>>>>>> or directory
>>>>>>>>>>>> tar: /etc/squid/squid.conf.pre.local: Cannot stat: No such 
>>>>>>>>>>>> file or directory
>>>>>>>>>>>>
>>>>>>>>>>>> There were seven lines in total like this and the three I 
>>>>>>>>>>>> checked were files that did not exist on my system. I 
>>>>>>>>>>>> presume that these are files that should be backed up if 
>>>>>>>>>>>> present but don't have to be present so that the error 
>>>>>>>>>>>> messages are nothing to worry about.
>>>>>>>>>>>> There were no error messages when doing the backup via the WUI.
>>>>>>>>>>>>
>>>>>>>>>>> Good find. ;)
>>>>>>>>>>> Michael's patch removes the test of existence, globbing 
>>>>>>>>>>> produces existent file names only. But I suppose there are 
>>>>>>>>>>> several full names of files, not existing in each 
>>>>>>>>>>> implementation.
>>>>>>>>>>> Conclusion: Repair the file globbing by the shopt and do the 
>>>>>>>>>>> existence check. So all existent files are chosen, that are 
>>>>>>>>>>> listed in the include files.
>>>>>>>>>>> Regards,
>>>>>>>>>>> Bernhard
>>>>>>>>>>>> The blank line was only on my vm testbed system but as I 
>>>>>>>>>>>> clone this for any testing activities I will have to check 
>>>>>>>>>>>> all of my existing vm's to fix this.
>>>>>>>>>>>>
>>>>>>>>>>>> Glad we resolved this simply. Sorry for the anxiety.
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>>
>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Am 29.03.2022 um 16:14 schrieb Adolf Belka:
>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>> This is what I get with the pushd/popd version in place 
>>>>>>>>>>>>>>> on the same system
>>>>>>>>>>>>>>> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
>>>>>>>>>>>>>>> etc/group
>>>>>>>>>>>>>>> etc/hosts
>>>>>>>>>>>>>>> etc/hosts.allow
>>>>>>>>>>>>>>> etc/hosts.deny
>>>>>>>>>>>>>>> etc/httpd/server.crt
>>>>>>>>>>>>>>> etc/httpd/server.csr
>>>>>>>>>>>>>>> etc/httpd/server-ecdsa.crt
>>>>>>>>>>>>>>> etc/httpd/server-ecdsa.csr
>>>>>>>>>>>>>>> etc/httpd/server-ecdsa.key
>>>>>>>>>>>>>>> etc/httpd/server.key
>>>>>>>>>>>>>>> etc/ipsec.user.conf
>>>>>>>>>>>>>>> etc/ipsec.user.secrets
>>>>>>>>>>>>>>> etc/logrotate.d
>>>>>>>>>>>>>>> etc/passwd
>>>>>>>>>>>>>>> etc/shadow
>>>>>>>>>>>>>>> etc/ssh/sshd_config
>>>>>>>>>>>>>>> etc/ssh/ssh_host_ecdsa_key
>>>>>>>>>>>>>>> etc/ssh/ssh_host_ecdsa_key.pub
>>>>>>>>>>>>>>> etc/ssh/ssh_host_ed25519_key
>>>>>>>>>>>>>>> etc/ssh/ssh_host_ed25519_key.pub
>>>>>>>>>>>>>>> etc/ssh/ssh_host_rsa_key
>>>>>>>>>>>>>>> etc/ssh/ssh_host_rsa_key.pub
>>>>>>>>>>>>>>> /etc/sudoers
>>>>>>>>>>>>>>> etc/sysconfig/createfiles
>>>>>>>>>>>>>>> etc/sysconfig/firewall.local
>>>>>>>>>>>>>>> etc/sysconfig/lm_sensors
>>>>>>>>>>>>>>> etc/sysconfig/modules
>>>>>>>>>>>>>>> etc/sysconfig/ramdisk
>>>>>>>>>>>>>>> etc/sysconfig/rc
>>>>>>>>>>>>>>> etc/sysconfig/rc.local
>>>>>>>>>>>>>>> etc/unbound
>>>>>>>>>>>>>>> /home/ahb
>>>>>>>>>>>>>>> root/.bash_history
>>>>>>>>>>>>>>> var/ipfire/accounting/settings.conf
>>>>>>>>>>>>>>> var/ipfire/auth/users
>>>>>>>>>>>>>>> var/ipfire/backup/addons/backup
>>>>>>>>>>>>>>> var/ipfire/backup/exclude.user
>>>>>>>>>>>>>>> var/ipfire/backup/include.user
>>>>>>>>>>>>>>> var/ipfire/ca/cacert.pem
>>>>>>>>>>>>>>> var/ipfire/captive/agb.txt
>>>>>>>>>>>>>>> var/ipfire/captive/clients
>>>>>>>>>>>>>>> var/ipfire/captive/coupons
>>>>>>>>>>>>>>> var/ipfire/captive/logo.dat
>>>>>>>>>>>>>>> var/ipfire/captive/settings
>>>>>>>>>>>>>>> var/ipfire/captive/terms.txt
>>>>>>>>>>>>>>> var/ipfire/captive/voucher_out
>>>>>>>>>>>>>>> var/ipfire/certs/hostcert.pem
>>>>>>>>>>>>>>> var/ipfire/certs/hostkey.pem
>>>>>>>>>>>>>>> var/ipfire/certs/phoebevmipseccert.pem
>>>>>>>>>>>>>>> var/ipfire/connscheduler/connscheduler.conf
>>>>>>>>>>>>>>> var/ipfire/crls/cacrl.pem
>>>>>>>>>>>>>>> var/ipfire/cups/cups-browsed.conf
>>>>>>>>>>>>>>> var/ipfire/cups/subscriptions.conf
>>>>>>>>>>>>>>> var/ipfire/ddns/config
>>>>>>>>>>>>>>> var/ipfire/ddns/ddns.conf
>>>>>>>>>>>>>>> var/ipfire/ddns/settings
>>>>>>>>>>>>>>> var/ipfire/dhcp/advoptions
>>>>>>>>>>>>>>> var/ipfire/dhcp/advoptions-list
>>>>>>>>>>>>>>> var/ipfire/dhcpc/dhcpcd.conf
>>>>>>>>>>>>>>> var/ipfire/dhcp/dhcpd.conf
>>>>>>>>>>>>>>> var/ipfire/dhcp/dhcpd.conf.local
>>>>>>>>>>>>>>> var/ipfire/dhcp/enable_blue
>>>>>>>>>>>>>>> var/ipfire/dhcp/enable_green
>>>>>>>>>>>>>>> var/ipfire/dhcp/fixleases
>>>>>>>>>>>>>>> var/ipfire/dhcp/settings
>>>>>>>>>>>>>>> var/ipfire/dma/auth.conf
>>>>>>>>>>>>>>> var/ipfire/dma/dma.conf
>>>>>>>>>>>>>>> var/ipfire/dma/mail.conf
>>>>>>>>>>>>>>> var/ipfire/dns
>>>>>>>>>>>>>>> var/ipfire/dnsforward/config
>>>>>>>>>>>>>>> var/ipfire/dns/settings
>>>>>>>>>>>>>>> var/ipfire/ethernet/aliases
>>>>>>>>>>>>>>> var/ipfire/ethernet/settings
>>>>>>>>>>>>>>> var/ipfire/ethernet/wireless
>>>>>>>>>>>>>>> var/ipfire/extrahd/settings
>>>>>>>>>>>>>>> var/ipfire/firewall
>>>>>>>>>>>>>>> var/ipfire/firewall/config
>>>>>>>>>>>>>>> var/ipfire/firewall/settings
>>>>>>>>>>>>>>> var/ipfire/fwhosts
>>>>>>>>>>>>>>> var/ipfire/isdn/settings
>>>>>>>>>>>>>>> var/ipfire/logging/settings
>>>>>>>>>>>>>>> var/ipfire/mac/settings
>>>>>>>>>>>>>>> var/ipfire/main/firstsetup_ok
>>>>>>>>>>>>>>> var/ipfire/main/gpl_accepted
>>>>>>>>>>>>>>> var/ipfire/main/hostname.conf
>>>>>>>>>>>>>>> var/ipfire/main/hosts
>>>>>>>>>>>>>>> var/ipfire/main/manualpages
>>>>>>>>>>>>>>> var/ipfire/main/routing
>>>>>>>>>>>>>>> var/ipfire/main/security
>>>>>>>>>>>>>>> var/ipfire/main/send_profile
>>>>>>>>>>>>>>> var/ipfire/main/settings
>>>>>>>>>>>>>>> var/ipfire/modem/settings
>>>>>>>>>>>>>>> var/ipfire/optionsfw/settings
>>>>>>>>>>>>>>> var/ipfire/ovpn
>>>>>>>>>>>>>>> var/ipfire/ovpn/ccd.conf
>>>>>>>>>>>>>>> var/ipfire/ovpn/collectd.vpn
>>>>>>>>>>>>>>> var/ipfire/ovpn/enable
>>>>>>>>>>>>>>> var/ipfire/ovpn/server.conf
>>>>>>>>>>>>>>> var/ipfire/ovpn/settings
>>>>>>>>>>>>>>> var/ipfire/pakfire/settings
>>>>>>>>>>>>>>> var/ipfire/ppp
>>>>>>>>>>>>>>> var/ipfire/ppp/fake-resolv.conf
>>>>>>>>>>>>>>> var/ipfire/ppp/settings
>>>>>>>>>>>>>>> var/ipfire/private/cakey.pem
>>>>>>>>>>>>>>> var/ipfire/proxy
>>>>>>>>>>>>>>> var/ipfire/proxy/asnbl-helper.conf
>>>>>>>>>>>>>>> var/ipfire/proxy/cachemgr.conf
>>>>>>>>>>>>>>> var/ipfire/proxy/enable
>>>>>>>>>>>>>>> var/ipfire/proxy/settings
>>>>>>>>>>>>>>> var/ipfire/proxy/squid.conf
>>>>>>>>>>>>>>> var/ipfire/qos/bin
>>>>>>>>>>>>>>> var/ipfire/qos/classes
>>>>>>>>>>>>>>> var/ipfire/qos/level7config
>>>>>>>>>>>>>>> var/ipfire/qos/portconfig
>>>>>>>>>>>>>>> var/ipfire/qos/settings
>>>>>>>>>>>>>>> var/ipfire/qos/subclasses
>>>>>>>>>>>>>>> var/ipfire/qos/tosconfig
>>>>>>>>>>>>>>> var/ipfire/remote/enablessh
>>>>>>>>>>>>>>> var/ipfire/remote/settings
>>>>>>>>>>>>>>> var/ipfire/sensors/settings
>>>>>>>>>>>>>>> var/ipfire/suricata/oinkmaster.conf
>>>>>>>>>>>>>>> var/ipfire/suricata/oinkmaster-modify-sids.conf
>>>>>>>>>>>>>>> var/ipfire/suricata/oinkmaster-provider-includes.conf
>>>>>>>>>>>>>>> var/ipfire/suricata/providers-settings
>>>>>>>>>>>>>>> var/ipfire/suricata/settings
>>>>>>>>>>>>>>> var/ipfire/suricata/suricata-default-rules.yaml
>>>>>>>>>>>>>>> var/ipfire/suricata/suricata-dns-servers.yaml
>>>>>>>>>>>>>>> var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
>>>>>>>>>>>>>>> var/ipfire/suricata/suricata-homenet.yaml
>>>>>>>>>>>>>>> var/ipfire/suricata/suricata-http-ports.yaml
>>>>>>>>>>>>>>> var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml 
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> var/ipfire/suricata/suricata-used-providers.yaml
>>>>>>>>>>>>>>> var/ipfire/time/
>>>>>>>>>>>>>>> var/ipfire/time/counter.conf
>>>>>>>>>>>>>>> var/ipfire/time/enable
>>>>>>>>>>>>>>> var/ipfire/time/settime.conf
>>>>>>>>>>>>>>> var/ipfire/time/settings
>>>>>>>>>>>>>>> var/ipfire/upnp/settings
>>>>>>>>>>>>>>> var/ipfire/urlfilter
>>>>>>>>>>>>>>> var/ipfire/urlfilter/settings
>>>>>>>>>>>>>>> var/ipfire/urlfilter/squidGuard.conf
>>>>>>>>>>>>>>> var/ipfire/vpn
>>>>>>>>>>>>>>> var/ipfire/vpn/config
>>>>>>>>>>>>>>> var/ipfire/vpn/ipsec.conf
>>>>>>>>>>>>>>> var/ipfire/vpn/settings
>>>>>>>>>>>>>>> var/ipfire/wakeonlan/clients.conf
>>>>>>>>>>>>>>> var/ipfire/wio/wio.conf
>>>>>>>>>>>>>>> var/ipfire/wireless/config
>>>>>>>>>>>>>>> var/ipfire/wireless/settings
>>>>>>>>>>>>>>> var/lib/suricata
>>>>>>>>>>>>>>> var/log/rrd/collectd
>>>>>>>>>>>>>>> var/log/rrd/hddshutdown-md127.rrd
>>>>>>>>>>>>>>> var/log/rrd/hddshutdown-sda.rrd
>>>>>>>>>>>>>>> var/log/rrd/hddshutdown-sdb.rrd
>>>>>>>>>>>>>>> var/log/rrd/hddtemp-md127.rrd
>>>>>>>>>>>>>>> var/log/rrd/hddtemp-sda.rrd
>>>>>>>>>>>>>>> var/log/rrd/hddtemp-sdb.rrd
>>>>>>>>>>>>>>> var/log/rrd/wio
>>>>>>>>>>>>>>> var/log/vnstat
>>>>>>>>>>>>>>> var/tmp/idsrules-emerging.tar.gz
>>>>>>>>>>>>>>> var/tmp/idsrules-sslbl_blacklist.rules
>>>>>>>>>>>>>>> The following are in the previous list but not in this one:-
>>>>>>>>>>>>>>> /
>>>>>>>>>>>>>>> /etc/conntrackd/conntrackd.conf
>>>>>>>>>>>>>>> /etc/ipsec.user-post.conf
>>>>>>>>>>>>>>> /root/.gitconfig
>>>>>>>>>>>>>>> /root/.ssh
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>> On 29/03/2022 16:01, Adolf Belka wrote:
>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 29/03/2022 15:36, Bernhard Bitsch wrote:
>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Am 29.03.2022 um 15:11 schrieb Michael Tremer:
>>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Could you please send the file listing to find out 
>>>>>>>>>>>>>>>>>> what is being included what shouldn’t?
>>>>>>>>>>>>>>>> Unfortunately, as I stopped the backup continuing once 
>>>>>>>>>>>>>>>> it had reached 1.2GB, the file created was not able to 
>>>>>>>>>>>>>>>> be opened. Probably stopping the backup corrupted it in 
>>>>>>>>>>>>>>>> some way.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Just for easy quick handling, you can use
>>>>>>>>>>>>>>>>> '/var/ipfire/backup/bin/backup.pl list' to show the 
>>>>>>>>>>>>>>>>> files included.
>>>>>>>>>>>>>>>> I gave this a go and here is the output from the command.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
>>>>>>>>>>>>>>>> /
>>>>>>>>>>>>>>>> /etc/conntrackd/conntrackd.conf
>>>>>>>>>>>>>>>> /etc/group
>>>>>>>>>>>>>>>> /etc/hosts
>>>>>>>>>>>>>>>> /etc/hosts.allow
>>>>>>>>>>>>>>>> /etc/hosts.deny
>>>>>>>>>>>>>>>> /etc/httpd/server.crt
>>>>>>>>>>>>>>>> /etc/httpd/server.csr
>>>>>>>>>>>>>>>> /etc/httpd/server-ecdsa.crt
>>>>>>>>>>>>>>>> /etc/httpd/server-ecdsa.csr
>>>>>>>>>>>>>>>> /etc/httpd/server-ecdsa.key
>>>>>>>>>>>>>>>> /etc/httpd/server.key
>>>>>>>>>>>>>>>> /etc/ipsec.user.conf
>>>>>>>>>>>>>>>> /etc/ipsec.user-post.conf
>>>>>>>>>>>>>>>> /etc/ipsec.user.secrets
>>>>>>>>>>>>>>>> /etc/logrotate.d
>>>>>>>>>>>>>>>> /etc/passwd
>>>>>>>>>>>>>>>> /etc/shadow
>>>>>>>>>>>>>>>> /etc/squid/squid.conf.local
>>>>>>>>>>>>>>>> /etc/squid/squid.conf.pre.local
>>>>>>>>>>>>>>>> /etc/ssh/sshd_config
>>>>>>>>>>>>>>>> /etc/ssh/ssh_host_ecdsa_key
>>>>>>>>>>>>>>>> /etc/ssh/ssh_host_ecdsa_key.pub
>>>>>>>>>>>>>>>> /etc/ssh/ssh_host_ed25519_key
>>>>>>>>>>>>>>>> /etc/ssh/ssh_host_ed25519_key.pub
>>>>>>>>>>>>>>>> /etc/ssh/ssh_host_rsa_key
>>>>>>>>>>>>>>>> /etc/ssh/ssh_host_rsa_key.pub
>>>>>>>>>>>>>>>> //etc/sudoers
>>>>>>>>>>>>>>>> /etc/sysconfig/createfiles
>>>>>>>>>>>>>>>> /etc/sysconfig/firewall.local
>>>>>>>>>>>>>>>> /etc/sysconfig/lm_sensors
>>>>>>>>>>>>>>>> /etc/sysconfig/modules
>>>>>>>>>>>>>>>> /etc/sysconfig/ramdisk
>>>>>>>>>>>>>>>> /etc/sysconfig/rc
>>>>>>>>>>>>>>>> /etc/sysconfig/rc.local
>>>>>>>>>>>>>>>> /etc/unbound
>>>>>>>>>>>>>>>> //home/ahb
>>>>>>>>>>>>>>>> /root/.bash_history
>>>>>>>>>>>>>>>> /root/.gitconfig
>>>>>>>>>>>>>>>> /root/.ssh
>>>>>>>>>>>>>>>> /var/ipfire/accounting/settings.conf
>>>>>>>>>>>>>>>> /var/ipfire/auth/users
>>>>>>>>>>>>>>>> /var/ipfire/backup/addons/backup
>>>>>>>>>>>>>>>> /var/ipfire/backup/exclude.user
>>>>>>>>>>>>>>>> /var/ipfire/backup/include.user
>>>>>>>>>>>>>>>> /var/ipfire/ca/cacert.pem
>>>>>>>>>>>>>>>> /var/ipfire/captive/agb.txt
>>>>>>>>>>>>>>>> /var/ipfire/captive/clients
>>>>>>>>>>>>>>>> /var/ipfire/captive/coupons
>>>>>>>>>>>>>>>> /var/ipfire/captive/logo.dat
>>>>>>>>>>>>>>>> /var/ipfire/captive/settings
>>>>>>>>>>>>>>>> /var/ipfire/captive/terms.txt
>>>>>>>>>>>>>>>> /var/ipfire/captive/voucher_out
>>>>>>>>>>>>>>>> /var/ipfire/certs/hostcert.pem
>>>>>>>>>>>>>>>> /var/ipfire/certs/hostkey.pem
>>>>>>>>>>>>>>>> /var/ipfire/certs/phoebevmipseccert.pem
>>>>>>>>>>>>>>>> /var/ipfire/connscheduler/connscheduler.conf
>>>>>>>>>>>>>>>> /var/ipfire/crls/cacrl.pem
>>>>>>>>>>>>>>>> /var/ipfire/cups/cups-browsed.conf
>>>>>>>>>>>>>>>> /var/ipfire/cups/subscriptions.conf
>>>>>>>>>>>>>>>> /var/ipfire/ddns/config
>>>>>>>>>>>>>>>> /var/ipfire/ddns/ddns.conf
>>>>>>>>>>>>>>>> /var/ipfire/ddns/settings
>>>>>>>>>>>>>>>> /var/ipfire/dhcp/advoptions
>>>>>>>>>>>>>>>> /var/ipfire/dhcp/advoptions-list
>>>>>>>>>>>>>>>> /var/ipfire/dhcpc/dhcpcd.conf
>>>>>>>>>>>>>>>> /var/ipfire/dhcp/dhcpd.conf
>>>>>>>>>>>>>>>> /var/ipfire/dhcp/dhcpd.conf.local
>>>>>>>>>>>>>>>> /var/ipfire/dhcp/enable_blue
>>>>>>>>>>>>>>>> /var/ipfire/dhcp/enable_green
>>>>>>>>>>>>>>>> /var/ipfire/dhcp/fixleases
>>>>>>>>>>>>>>>> /var/ipfire/dhcp/settings
>>>>>>>>>>>>>>>> /var/ipfire/dma/auth.conf
>>>>>>>>>>>>>>>> /var/ipfire/dma/dma.conf
>>>>>>>>>>>>>>>> /var/ipfire/dma/mail.conf
>>>>>>>>>>>>>>>> /var/ipfire/dns
>>>>>>>>>>>>>>>> /var/ipfire/dnsforward/config
>>>>>>>>>>>>>>>> /var/ipfire/dns/settings
>>>>>>>>>>>>>>>> /var/ipfire/ethernet/aliases
>>>>>>>>>>>>>>>> /var/ipfire/ethernet/settings
>>>>>>>>>>>>>>>> /var/ipfire/ethernet/wireless
>>>>>>>>>>>>>>>> /var/ipfire/extrahd/settings
>>>>>>>>>>>>>>>> /var/ipfire/firewall
>>>>>>>>>>>>>>>> /var/ipfire/firewall/config
>>>>>>>>>>>>>>>> /var/ipfire/firewall/settings
>>>>>>>>>>>>>>>> /var/ipfire/fwhosts
>>>>>>>>>>>>>>>> /var/ipfire/isdn/settings
>>>>>>>>>>>>>>>> /var/ipfire/logging/settings
>>>>>>>>>>>>>>>> /var/ipfire/mac/settings
>>>>>>>>>>>>>>>> /var/ipfire/main/firstsetup_ok
>>>>>>>>>>>>>>>> /var/ipfire/main/gpl_accepted
>>>>>>>>>>>>>>>> /var/ipfire/main/hostname.conf
>>>>>>>>>>>>>>>> /var/ipfire/main/hosts
>>>>>>>>>>>>>>>> /var/ipfire/main/manualpages
>>>>>>>>>>>>>>>> /var/ipfire/main/routing
>>>>>>>>>>>>>>>> /var/ipfire/main/security
>>>>>>>>>>>>>>>> /var/ipfire/main/send_profile
>>>>>>>>>>>>>>>> /var/ipfire/main/settings
>>>>>>>>>>>>>>>> /var/ipfire/modem/settings
>>>>>>>>>>>>>>>> /var/ipfire/optionsfw/settings
>>>>>>>>>>>>>>>> /var/ipfire/ovpn
>>>>>>>>>>>>>>>> /var/ipfire/ovpn/ccd.conf
>>>>>>>>>>>>>>>> /var/ipfire/ovpn/collectd.vpn
>>>>>>>>>>>>>>>> /var/ipfire/ovpn/enable
>>>>>>>>>>>>>>>> /var/ipfire/ovpn/server.conf
>>>>>>>>>>>>>>>> /var/ipfire/ovpn/settings
>>>>>>>>>>>>>>>> /var/ipfire/pakfire/settings
>>>>>>>>>>>>>>>> /var/ipfire/ppp
>>>>>>>>>>>>>>>> /var/ipfire/ppp/fake-resolv.conf
>>>>>>>>>>>>>>>> /var/ipfire/ppp/settings
>>>>>>>>>>>>>>>> /var/ipfire/private/cakey.pem
>>>>>>>>>>>>>>>> /var/ipfire/proxy
>>>>>>>>>>>>>>>> /var/ipfire/proxy/asnbl-helper.conf
>>>>>>>>>>>>>>>> /var/ipfire/proxy/cachemgr.conf
>>>>>>>>>>>>>>>> /var/ipfire/proxy/enable
>>>>>>>>>>>>>>>> /var/ipfire/proxy/settings
>>>>>>>>>>>>>>>> /var/ipfire/proxy/squid.conf
>>>>>>>>>>>>>>>> /var/ipfire/qos/bin
>>>>>>>>>>>>>>>> /var/ipfire/qos/bin/qos.sh
>>>>>>>>>>>>>>>> /var/ipfire/qos/classes
>>>>>>>>>>>>>>>> /var/ipfire/qos/level7config
>>>>>>>>>>>>>>>> /var/ipfire/qos/portconfig
>>>>>>>>>>>>>>>> /var/ipfire/qos/settings
>>>>>>>>>>>>>>>> /var/ipfire/qos/subclasses
>>>>>>>>>>>>>>>> /var/ipfire/qos/tosconfig
>>>>>>>>>>>>>>>> /var/ipfire/remote/enablessh
>>>>>>>>>>>>>>>> /var/ipfire/remote/settings
>>>>>>>>>>>>>>>> /var/ipfire/sensors/settings
>>>>>>>>>>>>>>>> /var/ipfire/suricata/oinkmaster.conf
>>>>>>>>>>>>>>>> /var/ipfire/suricata/oinkmaster-modify-sids.conf
>>>>>>>>>>>>>>>> /var/ipfire/suricata/oinkmaster-provider-includes.conf
>>>>>>>>>>>>>>>> /var/ipfire/suricata/providers-settings
>>>>>>>>>>>>>>>> /var/ipfire/suricata/settings
>>>>>>>>>>>>>>>> /var/ipfire/suricata/suricata-default-rules.yaml
>>>>>>>>>>>>>>>> /var/ipfire/suricata/suricata-dns-servers.yaml
>>>>>>>>>>>>>>>> /var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
>>>>>>>>>>>>>>>> /var/ipfire/suricata/suricata-homenet.yaml
>>>>>>>>>>>>>>>> /var/ipfire/suricata/suricata-http-ports.yaml
>>>>>>>>>>>>>>>> /var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml 
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> /var/ipfire/suricata/suricata-used-providers.yaml
>>>>>>>>>>>>>>>> /var/ipfire/time/
>>>>>>>>>>>>>>>> /var/ipfire/time/counter.conf
>>>>>>>>>>>>>>>> /var/ipfire/time/enable
>>>>>>>>>>>>>>>> /var/ipfire/time/settime.conf
>>>>>>>>>>>>>>>> /var/ipfire/time/settings
>>>>>>>>>>>>>>>> /var/ipfire/upnp/settings
>>>>>>>>>>>>>>>> /var/ipfire/urlfilter
>>>>>>>>>>>>>>>> /var/ipfire/urlfilter/settings
>>>>>>>>>>>>>>>> /var/ipfire/urlfilter/squidGuard.conf
>>>>>>>>>>>>>>>> /var/ipfire/vpn
>>>>>>>>>>>>>>>> /var/ipfire/vpn/config
>>>>>>>>>>>>>>>> /var/ipfire/vpn/ipsec.conf
>>>>>>>>>>>>>>>> /var/ipfire/vpn/settings
>>>>>>>>>>>>>>>> /var/ipfire/wakeonlan/clients.conf
>>>>>>>>>>>>>>>> /var/ipfire/wio/wio.conf
>>>>>>>>>>>>>>>> /var/ipfire/wireless/config
>>>>>>>>>>>>>>>> /var/ipfire/wireless/settings
>>>>>>>>>>>>>>>> /var/lib/suricata
>>>>>>>>>>>>>>>> /var/log/rrd/collectd
>>>>>>>>>>>>>>>> /var/log/rrd/hddshutdown-md127.rrd
>>>>>>>>>>>>>>>> /var/log/rrd/hddshutdown-sda.rrd
>>>>>>>>>>>>>>>> /var/log/rrd/hddshutdown-sdb.rrd
>>>>>>>>>>>>>>>> /var/log/rrd/hddtemp-md127.rrd
>>>>>>>>>>>>>>>> /var/log/rrd/hddtemp-sda.rrd
>>>>>>>>>>>>>>>> /var/log/rrd/hddtemp-sdb.rrd
>>>>>>>>>>>>>>>> /var/log/rrd/wio
>>>>>>>>>>>>>>>> /var/log/vnstat
>>>>>>>>>>>>>>>> /var/tmp/idsrules-emerging.tar.gz
>>>>>>>>>>>>>>>> /var/tmp/idsrules-sslbl_blacklist.rules
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Bernhard
>>>>>>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On 29 Mar 2022, at 14:10, Adolf Belka 
>>>>>>>>>>>>>>>>>>> <adolf.belka@ipfire.org> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Just tried this patch out on my vm testbed system and 
>>>>>>>>>>>>>>>>>>> it still doesn't work for me. The backup file had got 
>>>>>>>>>>>>>>>>>>> to 1.3GB when I deleted the backup file as it was 
>>>>>>>>>>>>>>>>>>> still growing. The normal correct backup file on that 
>>>>>>>>>>>>>>>>>>> vm machine is around 7MB
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> The above was the case for both running it from the 
>>>>>>>>>>>>>>>>>>> WUI or from the command line from my unprivileged 
>>>>>>>>>>>>>>>>>>> user using sudo backupctrl exclude
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On 29/03/2022 14:27, Michael Tremer wrote:
>>>>>>>>>>>>>>>>>>>> This patch fixes globbing expansion in the backup 
>>>>>>>>>>>>>>>>>>>> include file list
>>>>>>>>>>>>>>>>>>>> which got broken in 
>>>>>>>>>>>>>>>>>>>> c7e0d73e7cfd7be95db9d0a5f3392b8241813d5b.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Signed-off-by: Michael Tremer 
>>>>>>>>>>>>>>>>>>>> <michael.tremer@ipfire.org>
>>>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>>> config/backup/backup.pl | 8 ++++----
>>>>>>>>>>>>>>>>>>>> 1 file changed, 4 insertions(+), 4 deletions(-)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> diff --git a/config/backup/backup.pl 
>>>>>>>>>>>>>>>>>>>> b/config/backup/backup.pl
>>>>>>>>>>>>>>>>>>>> index a2337cf23..6f9295e94 100644
>>>>>>>>>>>>>>>>>>>> --- a/config/backup/backup.pl
>>>>>>>>>>>>>>>>>>>> +++ b/config/backup/backup.pl
>>>>>>>>>>>>>>>>>>>> @@ -19,6 +19,8 @@
>>>>>>>>>>>>>>>>>>>> # #
>>>>>>>>>>>>>>>>>>>> ############################################################################### 
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> +shopt -s nullglob
>>>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>>>> NOW="$(date "+%Y-%m-%d-%H:%M")"
>>>>>>>>>>>>>>>>>>>> list_addons() {
>>>>>>>>>>>>>>>>>>>> @@ -38,10 +40,8 @@ process_includes() {
>>>>>>>>>>>>>>>>>>>> for include in $@; do
>>>>>>>>>>>>>>>>>>>> local file
>>>>>>>>>>>>>>>>>>>> while read -r file; do
>>>>>>>>>>>>>>>>>>>> -  for file in ${file}; do
>>>>>>>>>>>>>>>>>>>> -  if [ -e "/${file}" ]; then
>>>>>>>>>>>>>>>>>>>> -  echo "${file}"
>>>>>>>>>>>>>>>>>>>> -  fi
>>>>>>>>>>>>>>>>>>>> +  for file in /${file}; do
>>>>>>>>>>>>>>>>>>>> +  echo "${file}"
>>>>>>>>>>>>>>>>>>>> done
>>>>>>>>>>>>>>>>>>>> done < "${include}"
>>>>>>>>>>>>>>>>>>>> done | sort -u
>>
>

Jon Murphy March 31, 2022, 7:27 p.m. UTC | #27

I was worried about the callable programs within the backup command.  (This is the error on 164 testing)

[root@ipfireVMc164t ~]# backupctrl restore /var/ipfire/backup/2022-03-17-14:05.ipf
. . .
/var/ipfire/backup/bin/backup.pl: line 151: convert-to-location: command not found
/var/ipfire/backup/bin/backup.pl: line 154: firewallctrl: command not found
/var/ipfire/backup/bin/backup.pl: line 157: convert-ovpn: command not found
/var/ipfire/backup/bin/backup.pl: line 175: convert-dns-settings: command not found
[root@ipfireVMc164t ~]# 

> On Mar 31, 2022, at 2:17 PM, Bernhard Bitsch <bbitsch@ipfire.org> wrote:
> 
> Hi,
> 
> I don't think so, also.
> 
> Bug #12811 shows a problem with the PATH variable ( for new installations ).
> The progs mentioned in bugzilla case should be callable.
> 
> Bug #12817 is about missing files in the backup. As Adolf stated, the programs should be found independent from the existence of the data files from the backup.
> 
> Regards,
> Bernhard
> 
> Am 31.03.2022 um 20:50 schrieb Adolf Belka:
>> Hi Jon,
>> On 31/03/2022 20:22, Jon Murphy wrote:
>>> Adolf,
>>> 
>>> Might this issue be part of the overall issue?  I am wondering if it was tested in the special release build?
>>> 
>>> https://bugzilla.ipfire.org/show_bug.cgi?id=12811 <https://bugzilla.ipfire.org/show_bug.cgi?id=12811>
>> I don't believe so. That bug already had a fix submitted into CU166 before bug 12817 had a fix for the broken globbing expansion and I did a quick test when CU166 was still very early on and it seemed to work OK.
>> This current bug 12817 regarding globbing expansion is to do with not finding the files to be backed up.
>> Bug 12811 is to do with the restore process and not the backup. After the files have been restored from the backup the backup.pl routine was not able to find some of the misc programs used to restart various things in IPFire after the restore. The fix was a modified default patch to include the location where those misc progs are located.
>> Having said all the above, I could also be wrong. Maybe someone else can say if my analysis is right or flawed.
>> Regards,
>> Adolf.
>>> 
>>> Jon
>>> 
>>>> On Mar 31, 2022, at 5:16 AM, Adolf Belka <adolf.belka@ipfire.org <mailto:adolf.belka@ipfire.org>> wrote:
>>>> 
>>>> Hi Michael,
>>>> 
>>>> Update on the testing. I realised that I had done the test on a CU175 version where I had modified the backup.pl code with pushd/popd.
>>>> I wouldn't expect that to give a problem but to confirm I created another CU165 clone with the backup.pl code in original form as most standard users would have it.
>>>> Re-ran the test and everything went correctly the same as my previous test.
>>>> 
>>>> Just for security I also did a fresh install of CU166 from the latest master nightly.
>>>> That also worked fine. The backup had all the directories as expected.
>>>> 
>>>> Regards,
>>>> Adolf
>>>> 
>>>> On 31/03/2022 09:46, Michael Tremer wrote:
>>>>> Hello Adolf,
>>>>> Brilliant. Thank you very much for testing this in detail.
>>>>> Best,
>>>>> -Michael
>>>>>> On 31 Mar 2022, at 08:41, Adolf Belka <adolf.belka@ipfire.org <mailto:adolf.belka@ipfire.org>> wrote:
>>>>>> 
>>>>>> Hi Michael,
>>>>>> 
>>>>>> On 30/03/2022 23:20, Adolf Belka wrote:
>>>>>>> Hi,
>>>>>>> 
>>>>>>> On 30/03/2022 17:05, Michael Tremer wrote:
>>>>>>>> Hello Bernhard,
>>>>>>>> 
>>>>>>>> You are obviously right. My brain is absolute mush these days and doesn’t allow me to function.
>>>>>>>> 
>>>>>>>> I will correct this by the end of the day.
>>>>>>>> 
>>>>>>>> -Michael
>>>>>>>> 
>>>>>>>>> On 30 Mar 2022, at 15:38, Bernhard Bitsch <bbitsch@ipfire.org <mailto:bbitsch@ipfire.org>> wrote:
>>>>>>>>> 
>>>>>>>>> Hello all,
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Am 30.03.2022 um 16:06 schrieb Michael Tremer:
>>>>>>>>>> Hello everyone,
>>>>>>>>>> Thank you very much for helping me test this.
>>>>>>>>>> I decided that we will have to create an emergency core update for this, because it will otherwise break people’s system. We also have a vulnerability in zlib which is being handed around by the press, so that has helped me to make a decision.
>>>>>>>>> 
>>>>>>>>> I think this is a good solution. Shall we add a warning to backup wiki page? Because in the moment a backup -> reinstall -> restore process leaves an incomplete system.
>>>>>>>>> 
>>>>>>>>>> I also pushed two more patches. One filters out any empty lines because they have quite a damaging effect and it happens easily that they are added to the include list. Secondly, I brought back the existence check which I thought was only there for the globbing mechanism, but also had the small side-effect of filtering out the file list early on which helps keeping tar happy.
>>>>>>>>>> Please review those changes and please install the new c166 from testing as soon as the build has finished (I am not sure if you are all on the nightly-builds list to see any notifications).
>>>>>>> I am not on the notifications list but I will check first thing tomorrow and when the latest has been updated I will do a test install and evaluation of the backup and provide confirmation of how it functions.
>>>>>>> 
>>>>>> As CU166 has been moved to master I took a clone of my CU165 vm and then did a CU on Testing with it rather than doing a fresh install. If you would also want a test of a fresh install done I can also do that.
>>>>>> 
>>>>>> The version I tested was IPFire 2.27 (x86_64) - Core Update 166 Development Build: master/8f696f60
>>>>>> 
>>>>>> Anyway the backup is working fine again. All the directories are present. The size of the backup is now very similar to previously also confirming all contents should be good.
>>>>>> 
>>>>>> I also added back in a couple of blank lines into my include.user file and the backup worked without any problems.
>>>>>> 
>>>>>> I also did a restore of the backup and again as far as I can tell with a quick check through on my testbed, everything is fine.
>>>>>> 
>>>>>> So everything looks good to me.
>>>>>> 
>>>>>> Regards,
>>>>>> 
>>>>>> Adolf.
>>>>>> 
>>>>>>> Regards,
>>>>>>> Adolf.
>>>>>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=4f0e7f24f293ca10096e616010cfd4417afc157c <https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=4f0e7f24f293ca10096e616010cfd4417afc157c> 
>>>>>>>>> The test on empty string should be in the 'while read -r file' loop. The empty lines are in the include files. The outer for loop iterates over the file list.
>>>>>>>>> 
>>>>>>>>> Regards,
>>>>>>>>> Bernhard
>>>>>>>>> 
>>>>>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=b275771fdd10ac7116e5d400dc10d3148df8ac27 <https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=b275771fdd10ac7116e5d400dc10d3148df8ac27> 
>>>>>>>>>> Best,
>>>>>>>>>> -Michael
>>>>>>>>>>> On 29 Mar 2022, at 20:21, Bernhard Bitsch <bbitsch@ipfire.org> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> Just checked the blank line problem.
>>>>>>>>>>> Pattern is empty , file name list is '/' --> no names produced ( with existence check )
>>>>>>>>>>> Pattern is empty --> produced file name '/' ( without -e check )
>>>>>>>>>>> 
>>>>>>>>>>> Am 29.03.2022 um 20:36 schrieb Bernhard Bitsch:
>>>>>>>>>>>> Hi all,
>>>>>>>>>>>> Am 29.03.2022 um 17:23 schrieb Adolf Belka:
>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On 29/03/2022 16:22, Michael Tremer wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On 29 Mar 2022, at 15:22, Bernhard Bitsch <bbitsch@ipfire.org> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Looks like your include.user file contains absolute paths.
>>>>>>>>>>>>> That was not the problem but I have corrected that. I would have expected that when the main include and exclude files were made relative that the update script would also have checked for any entries in the include.user and exclude.user files but it looks like it didn't.th
>>>>>>>>>>>> According to the source they are chosen.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Or an empty line maybe?
>>>>>>>>>>>>> That turned out to be the problem. There was an empty line after the two lines I had added. You only saw it if you scrolled the pointer down the file in the editor.
>>>>>>>>>>>> Interesting. A blank line should produce nothing.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Removing that blank line removed the line only with a / and the backup then successfully ran and created a backup file containing all the directories expected.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> The only thing I found is that there was the following tar error message
>>>>>>>>>>>>> 
>>>>>>>>>>>>> tar: Exiting with failure status due to previous errors
>>>>>>>>>>>>> 
>>>>>>>>>>>>> The only thing I found in the backup output was several lines such as
>>>>>>>>>>>>> 
>>>>>>>>>>>>> tar: /etc/squid/squid.conf.local: Cannot stat: No such file or directory
>>>>>>>>>>>>> tar: /etc/squid/squid.conf.pre.local: Cannot stat: No such file or directory
>>>>>>>>>>>>> 
>>>>>>>>>>>>> There were seven lines in total like this and the three I checked were files that did not exist on my system. I presume that these are files that should be backed up if present but don't have to be present so that the error messages are nothing to worry about.
>>>>>>>>>>>>> There were no error messages when doing the backup via the WUI.
>>>>>>>>>>>>> 
>>>>>>>>>>>> Good find. ;)
>>>>>>>>>>>> Michael's patch removes the test of existence, globbing produces existent file names only. But I suppose there are several full names of files, not existing in each implementation.
>>>>>>>>>>>> Conclusion: Repair the file globbing by the shopt and do the existence check. So all existent files are chosen, that are listed in the include files.
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Bernhard
>>>>>>>>>>>>> The blank line was only on my vm testbed system but as I clone this for any testing activities I will have to check all of my existing vm's to fix this.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Glad we resolved this simply. Sorry for the anxiety.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Am 29.03.2022 um 16:14 schrieb Adolf Belka:
>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>> This is what I get with the pushd/popd version in place on the same system
>>>>>>>>>>>>>>>> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
>>>>>>>>>>>>>>>> etc/group
>>>>>>>>>>>>>>>> etc/hosts
>>>>>>>>>>>>>>>> etc/hosts.allow
>>>>>>>>>>>>>>>> etc/hosts.deny
>>>>>>>>>>>>>>>> etc/httpd/server.crt
>>>>>>>>>>>>>>>> etc/httpd/server.csr
>>>>>>>>>>>>>>>> etc/httpd/server-ecdsa.crt
>>>>>>>>>>>>>>>> etc/httpd/server-ecdsa.csr
>>>>>>>>>>>>>>>> etc/httpd/server-ecdsa.key
>>>>>>>>>>>>>>>> etc/httpd/server.key
>>>>>>>>>>>>>>>> etc/ipsec.user.conf
>>>>>>>>>>>>>>>> etc/ipsec.user.secrets
>>>>>>>>>>>>>>>> etc/logrotate.d
>>>>>>>>>>>>>>>> etc/passwd
>>>>>>>>>>>>>>>> etc/shadow
>>>>>>>>>>>>>>>> etc/ssh/sshd_config
>>>>>>>>>>>>>>>> etc/ssh/ssh_host_ecdsa_key
>>>>>>>>>>>>>>>> etc/ssh/ssh_host_ecdsa_key.pub
>>>>>>>>>>>>>>>> etc/ssh/ssh_host_ed25519_key
>>>>>>>>>>>>>>>> etc/ssh/ssh_host_ed25519_key.pub
>>>>>>>>>>>>>>>> etc/ssh/ssh_host_rsa_key
>>>>>>>>>>>>>>>> etc/ssh/ssh_host_rsa_key.pub
>>>>>>>>>>>>>>>> /etc/sudoers
>>>>>>>>>>>>>>>> etc/sysconfig/createfiles
>>>>>>>>>>>>>>>> etc/sysconfig/firewall.local
>>>>>>>>>>>>>>>> etc/sysconfig/lm_sensors
>>>>>>>>>>>>>>>> etc/sysconfig/modules
>>>>>>>>>>>>>>>> etc/sysconfig/ramdisk
>>>>>>>>>>>>>>>> etc/sysconfig/rc
>>>>>>>>>>>>>>>> etc/sysconfig/rc.local
>>>>>>>>>>>>>>>> etc/unbound
>>>>>>>>>>>>>>>> /home/ahb
>>>>>>>>>>>>>>>> root/.bash_history
>>>>>>>>>>>>>>>> var/ipfire/accounting/settings.conf
>>>>>>>>>>>>>>>> var/ipfire/auth/users
>>>>>>>>>>>>>>>> var/ipfire/backup/addons/backup
>>>>>>>>>>>>>>>> var/ipfire/backup/exclude.user
>>>>>>>>>>>>>>>> var/ipfire/backup/include.user
>>>>>>>>>>>>>>>> var/ipfire/ca/cacert.pem
>>>>>>>>>>>>>>>> var/ipfire/captive/agb.txt
>>>>>>>>>>>>>>>> var/ipfire/captive/clients
>>>>>>>>>>>>>>>> var/ipfire/captive/coupons
>>>>>>>>>>>>>>>> var/ipfire/captive/logo.dat
>>>>>>>>>>>>>>>> var/ipfire/captive/settings
>>>>>>>>>>>>>>>> var/ipfire/captive/terms.txt
>>>>>>>>>>>>>>>> var/ipfire/captive/voucher_out
>>>>>>>>>>>>>>>> var/ipfire/certs/hostcert.pem
>>>>>>>>>>>>>>>> var/ipfire/certs/hostkey.pem
>>>>>>>>>>>>>>>> var/ipfire/certs/phoebevmipseccert.pem
>>>>>>>>>>>>>>>> var/ipfire/connscheduler/connscheduler.conf
>>>>>>>>>>>>>>>> var/ipfire/crls/cacrl.pem
>>>>>>>>>>>>>>>> var/ipfire/cups/cups-browsed.conf
>>>>>>>>>>>>>>>> var/ipfire/cups/subscriptions.conf
>>>>>>>>>>>>>>>> var/ipfire/ddns/config
>>>>>>>>>>>>>>>> var/ipfire/ddns/ddns.conf
>>>>>>>>>>>>>>>> var/ipfire/ddns/settings
>>>>>>>>>>>>>>>> var/ipfire/dhcp/advoptions
>>>>>>>>>>>>>>>> var/ipfire/dhcp/advoptions-list
>>>>>>>>>>>>>>>> var/ipfire/dhcpc/dhcpcd.conf
>>>>>>>>>>>>>>>> var/ipfire/dhcp/dhcpd.conf
>>>>>>>>>>>>>>>> var/ipfire/dhcp/dhcpd.conf.local
>>>>>>>>>>>>>>>> var/ipfire/dhcp/enable_blue
>>>>>>>>>>>>>>>> var/ipfire/dhcp/enable_green
>>>>>>>>>>>>>>>> var/ipfire/dhcp/fixleases
>>>>>>>>>>>>>>>> var/ipfire/dhcp/settings
>>>>>>>>>>>>>>>> var/ipfire/dma/auth.conf
>>>>>>>>>>>>>>>> var/ipfire/dma/dma.conf
>>>>>>>>>>>>>>>> var/ipfire/dma/mail.conf
>>>>>>>>>>>>>>>> var/ipfire/dns
>>>>>>>>>>>>>>>> var/ipfire/dnsforward/config
>>>>>>>>>>>>>>>> var/ipfire/dns/settings
>>>>>>>>>>>>>>>> var/ipfire/ethernet/aliases
>>>>>>>>>>>>>>>> var/ipfire/ethernet/settings
>>>>>>>>>>>>>>>> var/ipfire/ethernet/wireless
>>>>>>>>>>>>>>>> var/ipfire/extrahd/settings
>>>>>>>>>>>>>>>> var/ipfire/firewall
>>>>>>>>>>>>>>>> var/ipfire/firewall/config
>>>>>>>>>>>>>>>> var/ipfire/firewall/settings
>>>>>>>>>>>>>>>> var/ipfire/fwhosts
>>>>>>>>>>>>>>>> var/ipfire/isdn/settings
>>>>>>>>>>>>>>>> var/ipfire/logging/settings
>>>>>>>>>>>>>>>> var/ipfire/mac/settings
>>>>>>>>>>>>>>>> var/ipfire/main/firstsetup_ok
>>>>>>>>>>>>>>>> var/ipfire/main/gpl_accepted
>>>>>>>>>>>>>>>> var/ipfire/main/hostname.conf
>>>>>>>>>>>>>>>> var/ipfire/main/hosts
>>>>>>>>>>>>>>>> var/ipfire/main/manualpages
>>>>>>>>>>>>>>>> var/ipfire/main/routing
>>>>>>>>>>>>>>>> var/ipfire/main/security
>>>>>>>>>>>>>>>> var/ipfire/main/send_profile
>>>>>>>>>>>>>>>> var/ipfire/main/settings
>>>>>>>>>>>>>>>> var/ipfire/modem/settings
>>>>>>>>>>>>>>>> var/ipfire/optionsfw/settings
>>>>>>>>>>>>>>>> var/ipfire/ovpn
>>>>>>>>>>>>>>>> var/ipfire/ovpn/ccd.conf
>>>>>>>>>>>>>>>> var/ipfire/ovpn/collectd.vpn
>>>>>>>>>>>>>>>> var/ipfire/ovpn/enable
>>>>>>>>>>>>>>>> var/ipfire/ovpn/server.conf
>>>>>>>>>>>>>>>> var/ipfire/ovpn/settings
>>>>>>>>>>>>>>>> var/ipfire/pakfire/settings
>>>>>>>>>>>>>>>> var/ipfire/ppp
>>>>>>>>>>>>>>>> var/ipfire/ppp/fake-resolv.conf
>>>>>>>>>>>>>>>> var/ipfire/ppp/settings
>>>>>>>>>>>>>>>> var/ipfire/private/cakey.pem
>>>>>>>>>>>>>>>> var/ipfire/proxy
>>>>>>>>>>>>>>>> var/ipfire/proxy/asnbl-helper.conf
>>>>>>>>>>>>>>>> var/ipfire/proxy/cachemgr.conf
>>>>>>>>>>>>>>>> var/ipfire/proxy/enable
>>>>>>>>>>>>>>>> var/ipfire/proxy/settings
>>>>>>>>>>>>>>>> var/ipfire/proxy/squid.conf
>>>>>>>>>>>>>>>> var/ipfire/qos/bin
>>>>>>>>>>>>>>>> var/ipfire/qos/classes
>>>>>>>>>>>>>>>> var/ipfire/qos/level7config
>>>>>>>>>>>>>>>> var/ipfire/qos/portconfig
>>>>>>>>>>>>>>>> var/ipfire/qos/settings
>>>>>>>>>>>>>>>> var/ipfire/qos/subclasses
>>>>>>>>>>>>>>>> var/ipfire/qos/tosconfig
>>>>>>>>>>>>>>>> var/ipfire/remote/enablessh
>>>>>>>>>>>>>>>> var/ipfire/remote/settings
>>>>>>>>>>>>>>>> var/ipfire/sensors/settings
>>>>>>>>>>>>>>>> var/ipfire/suricata/oinkmaster.conf
>>>>>>>>>>>>>>>> var/ipfire/suricata/oinkmaster-modify-sids.conf
>>>>>>>>>>>>>>>> var/ipfire/suricata/oinkmaster-provider-includes.conf
>>>>>>>>>>>>>>>> var/ipfire/suricata/providers-settings
>>>>>>>>>>>>>>>> var/ipfire/suricata/settings
>>>>>>>>>>>>>>>> var/ipfire/suricata/suricata-default-rules.yaml
>>>>>>>>>>>>>>>> var/ipfire/suricata/suricata-dns-servers.yaml
>>>>>>>>>>>>>>>> var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
>>>>>>>>>>>>>>>> var/ipfire/suricata/suricata-homenet.yaml
>>>>>>>>>>>>>>>> var/ipfire/suricata/suricata-http-ports.yaml
>>>>>>>>>>>>>>>> var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml 
>>>>>>>>>>>>>>>> var/ipfire/suricata/suricata-used-providers.yaml
>>>>>>>>>>>>>>>> var/ipfire/time/
>>>>>>>>>>>>>>>> var/ipfire/time/counter.conf
>>>>>>>>>>>>>>>> var/ipfire/time/enable
>>>>>>>>>>>>>>>> var/ipfire/time/settime.conf
>>>>>>>>>>>>>>>> var/ipfire/time/settings
>>>>>>>>>>>>>>>> var/ipfire/upnp/settings
>>>>>>>>>>>>>>>> var/ipfire/urlfilter
>>>>>>>>>>>>>>>> var/ipfire/urlfilter/settings
>>>>>>>>>>>>>>>> var/ipfire/urlfilter/squidGuard.conf
>>>>>>>>>>>>>>>> var/ipfire/vpn
>>>>>>>>>>>>>>>> var/ipfire/vpn/config
>>>>>>>>>>>>>>>> var/ipfire/vpn/ipsec.conf
>>>>>>>>>>>>>>>> var/ipfire/vpn/settings
>>>>>>>>>>>>>>>> var/ipfire/wakeonlan/clients.conf
>>>>>>>>>>>>>>>> var/ipfire/wio/wio.conf
>>>>>>>>>>>>>>>> var/ipfire/wireless/config
>>>>>>>>>>>>>>>> var/ipfire/wireless/settings
>>>>>>>>>>>>>>>> var/lib/suricata
>>>>>>>>>>>>>>>> var/log/rrd/collectd
>>>>>>>>>>>>>>>> var/log/rrd/hddshutdown-md127.rrd
>>>>>>>>>>>>>>>> var/log/rrd/hddshutdown-sda.rrd
>>>>>>>>>>>>>>>> var/log/rrd/hddshutdown-sdb.rrd
>>>>>>>>>>>>>>>> var/log/rrd/hddtemp-md127.rrd
>>>>>>>>>>>>>>>> var/log/rrd/hddtemp-sda.rrd
>>>>>>>>>>>>>>>> var/log/rrd/hddtemp-sdb.rrd
>>>>>>>>>>>>>>>> var/log/rrd/wio
>>>>>>>>>>>>>>>> var/log/vnstat
>>>>>>>>>>>>>>>> var/tmp/idsrules-emerging.tar.gz
>>>>>>>>>>>>>>>> var/tmp/idsrules-sslbl_blacklist.rules
>>>>>>>>>>>>>>>> The following are in the previous list but not in this one:-
>>>>>>>>>>>>>>>> /
>>>>>>>>>>>>>>>> /etc/conntrackd/conntrackd.conf
>>>>>>>>>>>>>>>> /etc/ipsec.user-post.conf
>>>>>>>>>>>>>>>> /root/.gitconfig
>>>>>>>>>>>>>>>> /root/.ssh
>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>> On 29/03/2022 16:01, Adolf Belka wrote:
>>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On 29/03/2022 15:36, Bernhard Bitsch wrote:
>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Am 29.03.2022 um 15:11 schrieb Michael Tremer:
>>>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Could you please send the file listing to find out what is being included what shouldn’t?
>>>>>>>>>>>>>>>>> Unfortunately, as I stopped the backup continuing once it had reached 1.2GB, the file created was not able to be opened. Probably stopping the backup corrupted it in some way.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Just for easy quick handling, you can use
>>>>>>>>>>>>>>>>>> '/var/ipfire/backup/bin/backup.pl list' to show the files included.
>>>>>>>>>>>>>>>>> I gave this a go and here is the output from the command.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
>>>>>>>>>>>>>>>>> /
>>>>>>>>>>>>>>>>> /etc/conntrackd/conntrackd.conf
>>>>>>>>>>>>>>>>> /etc/group
>>>>>>>>>>>>>>>>> /etc/hosts
>>>>>>>>>>>>>>>>> /etc/hosts.allow
>>>>>>>>>>>>>>>>> /etc/hosts.deny
>>>>>>>>>>>>>>>>> /etc/httpd/server.crt
>>>>>>>>>>>>>>>>> /etc/httpd/server.csr
>>>>>>>>>>>>>>>>> /etc/httpd/server-ecdsa.crt
>>>>>>>>>>>>>>>>> /etc/httpd/server-ecdsa.csr
>>>>>>>>>>>>>>>>> /etc/httpd/server-ecdsa.key
>>>>>>>>>>>>>>>>> /etc/httpd/server.key
>>>>>>>>>>>>>>>>> /etc/ipsec.user.conf
>>>>>>>>>>>>>>>>> /etc/ipsec.user-post.conf
>>>>>>>>>>>>>>>>> /etc/ipsec.user.secrets
>>>>>>>>>>>>>>>>> /etc/logrotate.d
>>>>>>>>>>>>>>>>> /etc/passwd
>>>>>>>>>>>>>>>>> /etc/shadow
>>>>>>>>>>>>>>>>> /etc/squid/squid.conf.local
>>>>>>>>>>>>>>>>> /etc/squid/squid.conf.pre.local
>>>>>>>>>>>>>>>>> /etc/ssh/sshd_config
>>>>>>>>>>>>>>>>> /etc/ssh/ssh_host_ecdsa_key
>>>>>>>>>>>>>>>>> /etc/ssh/ssh_host_ecdsa_key.pub
>>>>>>>>>>>>>>>>> /etc/ssh/ssh_host_ed25519_key
>>>>>>>>>>>>>>>>> /etc/ssh/ssh_host_ed25519_key.pub
>>>>>>>>>>>>>>>>> /etc/ssh/ssh_host_rsa_key
>>>>>>>>>>>>>>>>> /etc/ssh/ssh_host_rsa_key.pub
>>>>>>>>>>>>>>>>> //etc/sudoers
>>>>>>>>>>>>>>>>> /etc/sysconfig/createfiles
>>>>>>>>>>>>>>>>> /etc/sysconfig/firewall.local
>>>>>>>>>>>>>>>>> /etc/sysconfig/lm_sensors
>>>>>>>>>>>>>>>>> /etc/sysconfig/modules
>>>>>>>>>>>>>>>>> /etc/sysconfig/ramdisk
>>>>>>>>>>>>>>>>> /etc/sysconfig/rc
>>>>>>>>>>>>>>>>> /etc/sysconfig/rc.local
>>>>>>>>>>>>>>>>> /etc/unbound
>>>>>>>>>>>>>>>>> //home/ahb
>>>>>>>>>>>>>>>>> /root/.bash_history
>>>>>>>>>>>>>>>>> /root/.gitconfig
>>>>>>>>>>>>>>>>> /root/.ssh
>>>>>>>>>>>>>>>>> /var/ipfire/accounting/settings.conf
>>>>>>>>>>>>>>>>> /var/ipfire/auth/users
>>>>>>>>>>>>>>>>> /var/ipfire/backup/addons/backup
>>>>>>>>>>>>>>>>> /var/ipfire/backup/exclude.user
>>>>>>>>>>>>>>>>> /var/ipfire/backup/include.user
>>>>>>>>>>>>>>>>> /var/ipfire/ca/cacert.pem
>>>>>>>>>>>>>>>>> /var/ipfire/captive/agb.txt
>>>>>>>>>>>>>>>>> /var/ipfire/captive/clients
>>>>>>>>>>>>>>>>> /var/ipfire/captive/coupons
>>>>>>>>>>>>>>>>> /var/ipfire/captive/logo.dat
>>>>>>>>>>>>>>>>> /var/ipfire/captive/settings
>>>>>>>>>>>>>>>>> /var/ipfire/captive/terms.txt
>>>>>>>>>>>>>>>>> /var/ipfire/captive/voucher_out
>>>>>>>>>>>>>>>>> /var/ipfire/certs/hostcert.pem
>>>>>>>>>>>>>>>>> /var/ipfire/certs/hostkey.pem
>>>>>>>>>>>>>>>>> /var/ipfire/certs/phoebevmipseccert.pem
>>>>>>>>>>>>>>>>> /var/ipfire/connscheduler/connscheduler.conf
>>>>>>>>>>>>>>>>> /var/ipfire/crls/cacrl.pem
>>>>>>>>>>>>>>>>> /var/ipfire/cups/cups-browsed.conf
>>>>>>>>>>>>>>>>> /var/ipfire/cups/subscriptions.conf
>>>>>>>>>>>>>>>>> /var/ipfire/ddns/config
>>>>>>>>>>>>>>>>> /var/ipfire/ddns/ddns.conf
>>>>>>>>>>>>>>>>> /var/ipfire/ddns/settings
>>>>>>>>>>>>>>>>> /var/ipfire/dhcp/advoptions
>>>>>>>>>>>>>>>>> /var/ipfire/dhcp/advoptions-list
>>>>>>>>>>>>>>>>> /var/ipfire/dhcpc/dhcpcd.conf
>>>>>>>>>>>>>>>>> /var/ipfire/dhcp/dhcpd.conf
>>>>>>>>>>>>>>>>> /var/ipfire/dhcp/dhcpd.conf.local
>>>>>>>>>>>>>>>>> /var/ipfire/dhcp/enable_blue
>>>>>>>>>>>>>>>>> /var/ipfire/dhcp/enable_green
>>>>>>>>>>>>>>>>> /var/ipfire/dhcp/fixleases
>>>>>>>>>>>>>>>>> /var/ipfire/dhcp/settings
>>>>>>>>>>>>>>>>> /var/ipfire/dma/auth.conf
>>>>>>>>>>>>>>>>> /var/ipfire/dma/dma.conf
>>>>>>>>>>>>>>>>> /var/ipfire/dma/mail.conf
>>>>>>>>>>>>>>>>> /var/ipfire/dns
>>>>>>>>>>>>>>>>> /var/ipfire/dnsforward/config
>>>>>>>>>>>>>>>>> /var/ipfire/dns/settings
>>>>>>>>>>>>>>>>> /var/ipfire/ethernet/aliases
>>>>>>>>>>>>>>>>> /var/ipfire/ethernet/settings
>>>>>>>>>>>>>>>>> /var/ipfire/ethernet/wireless
>>>>>>>>>>>>>>>>> /var/ipfire/extrahd/settings
>>>>>>>>>>>>>>>>> /var/ipfire/firewall
>>>>>>>>>>>>>>>>> /var/ipfire/firewall/config
>>>>>>>>>>>>>>>>> /var/ipfire/firewall/settings
>>>>>>>>>>>>>>>>> /var/ipfire/fwhosts
>>>>>>>>>>>>>>>>> /var/ipfire/isdn/settings
>>>>>>>>>>>>>>>>> /var/ipfire/logging/settings
>>>>>>>>>>>>>>>>> /var/ipfire/mac/settings
>>>>>>>>>>>>>>>>> /var/ipfire/main/firstsetup_ok
>>>>>>>>>>>>>>>>> /var/ipfire/main/gpl_accepted
>>>>>>>>>>>>>>>>> /var/ipfire/main/hostname.conf
>>>>>>>>>>>>>>>>> /var/ipfire/main/hosts
>>>>>>>>>>>>>>>>> /var/ipfire/main/manualpages
>>>>>>>>>>>>>>>>> /var/ipfire/main/routing
>>>>>>>>>>>>>>>>> /var/ipfire/main/security
>>>>>>>>>>>>>>>>> /var/ipfire/main/send_profile
>>>>>>>>>>>>>>>>> /var/ipfire/main/settings
>>>>>>>>>>>>>>>>> /var/ipfire/modem/settings
>>>>>>>>>>>>>>>>> /var/ipfire/optionsfw/settings
>>>>>>>>>>>>>>>>> /var/ipfire/ovpn
>>>>>>>>>>>>>>>>> /var/ipfire/ovpn/ccd.conf
>>>>>>>>>>>>>>>>> /var/ipfire/ovpn/collectd.vpn
>>>>>>>>>>>>>>>>> /var/ipfire/ovpn/enable
>>>>>>>>>>>>>>>>> /var/ipfire/ovpn/server.conf
>>>>>>>>>>>>>>>>> /var/ipfire/ovpn/settings
>>>>>>>>>>>>>>>>> /var/ipfire/pakfire/settings
>>>>>>>>>>>>>>>>> /var/ipfire/ppp
>>>>>>>>>>>>>>>>> /var/ipfire/ppp/fake-resolv.conf
>>>>>>>>>>>>>>>>> /var/ipfire/ppp/settings
>>>>>>>>>>>>>>>>> /var/ipfire/private/cakey.pem
>>>>>>>>>>>>>>>>> /var/ipfire/proxy
>>>>>>>>>>>>>>>>> /var/ipfire/proxy/asnbl-helper.conf
>>>>>>>>>>>>>>>>> /var/ipfire/proxy/cachemgr.conf
>>>>>>>>>>>>>>>>> /var/ipfire/proxy/enable
>>>>>>>>>>>>>>>>> /var/ipfire/proxy/settings
>>>>>>>>>>>>>>>>> /var/ipfire/proxy/squid.conf
>>>>>>>>>>>>>>>>> /var/ipfire/qos/bin
>>>>>>>>>>>>>>>>> /var/ipfire/qos/bin/qos.sh
>>>>>>>>>>>>>>>>> /var/ipfire/qos/classes
>>>>>>>>>>>>>>>>> /var/ipfire/qos/level7config
>>>>>>>>>>>>>>>>> /var/ipfire/qos/portconfig
>>>>>>>>>>>>>>>>> /var/ipfire/qos/settings
>>>>>>>>>>>>>>>>> /var/ipfire/qos/subclasses
>>>>>>>>>>>>>>>>> /var/ipfire/qos/tosconfig
>>>>>>>>>>>>>>>>> /var/ipfire/remote/enablessh
>>>>>>>>>>>>>>>>> /var/ipfire/remote/settings
>>>>>>>>>>>>>>>>> /var/ipfire/sensors/settings
>>>>>>>>>>>>>>>>> /var/ipfire/suricata/oinkmaster.conf
>>>>>>>>>>>>>>>>> /var/ipfire/suricata/oinkmaster-modify-sids.conf
>>>>>>>>>>>>>>>>> /var/ipfire/suricata/oinkmaster-provider-includes.conf
>>>>>>>>>>>>>>>>> /var/ipfire/suricata/providers-settings
>>>>>>>>>>>>>>>>> /var/ipfire/suricata/settings
>>>>>>>>>>>>>>>>> /var/ipfire/suricata/suricata-default-rules.yaml
>>>>>>>>>>>>>>>>> /var/ipfire/suricata/suricata-dns-servers.yaml
>>>>>>>>>>>>>>>>> /var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
>>>>>>>>>>>>>>>>> /var/ipfire/suricata/suricata-homenet.yaml
>>>>>>>>>>>>>>>>> /var/ipfire/suricata/suricata-http-ports.yaml
>>>>>>>>>>>>>>>>> /var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml 
>>>>>>>>>>>>>>>>> /var/ipfire/suricata/suricata-used-providers.yaml
>>>>>>>>>>>>>>>>> /var/ipfire/time/
>>>>>>>>>>>>>>>>> /var/ipfire/time/counter.conf
>>>>>>>>>>>>>>>>> /var/ipfire/time/enable
>>>>>>>>>>>>>>>>> /var/ipfire/time/settime.conf
>>>>>>>>>>>>>>>>> /var/ipfire/time/settings
>>>>>>>>>>>>>>>>> /var/ipfire/upnp/settings
>>>>>>>>>>>>>>>>> /var/ipfire/urlfilter
>>>>>>>>>>>>>>>>> /var/ipfire/urlfilter/settings
>>>>>>>>>>>>>>>>> /var/ipfire/urlfilter/squidGuard.conf
>>>>>>>>>>>>>>>>> /var/ipfire/vpn
>>>>>>>>>>>>>>>>> /var/ipfire/vpn/config
>>>>>>>>>>>>>>>>> /var/ipfire/vpn/ipsec.conf
>>>>>>>>>>>>>>>>> /var/ipfire/vpn/settings
>>>>>>>>>>>>>>>>> /var/ipfire/wakeonlan/clients.conf
>>>>>>>>>>>>>>>>> /var/ipfire/wio/wio.conf
>>>>>>>>>>>>>>>>> /var/ipfire/wireless/config
>>>>>>>>>>>>>>>>> /var/ipfire/wireless/settings
>>>>>>>>>>>>>>>>> /var/lib/suricata
>>>>>>>>>>>>>>>>> /var/log/rrd/collectd
>>>>>>>>>>>>>>>>> /var/log/rrd/hddshutdown-md127.rrd
>>>>>>>>>>>>>>>>> /var/log/rrd/hddshutdown-sda.rrd
>>>>>>>>>>>>>>>>> /var/log/rrd/hddshutdown-sdb.rrd
>>>>>>>>>>>>>>>>> /var/log/rrd/hddtemp-md127.rrd
>>>>>>>>>>>>>>>>> /var/log/rrd/hddtemp-sda.rrd
>>>>>>>>>>>>>>>>> /var/log/rrd/hddtemp-sdb.rrd
>>>>>>>>>>>>>>>>> /var/log/rrd/wio
>>>>>>>>>>>>>>>>> /var/log/vnstat
>>>>>>>>>>>>>>>>> /var/tmp/idsrules-emerging.tar.gz
>>>>>>>>>>>>>>>>> /var/tmp/idsrules-sslbl_blacklist.rules
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Bernhard
>>>>>>>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On 29 Mar 2022, at 14:10, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Just tried this patch out on my vm testbed system and it still doesn't work for me. The backup file had got to 1.3GB when I deleted the backup file as it was still growing. The normal correct backup file on that vm machine is around 7MB
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> The above was the case for both running it from the WUI or from the command line from my unprivileged user using sudo backupctrl exclude
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On 29/03/2022 14:27, Michael Tremer wrote:
>>>>>>>>>>>>>>>>>>>>> This patch fixes globbing expansion in the backup include file list
>>>>>>>>>>>>>>>>>>>>> which got broken in c7e0d73e7cfd7be95db9d0a5f3392b8241813d5b.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
>>>>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>>>> config/backup/bac