Message ID | 20220825003129.3552746-1-jon.murphy@ipfire.org |
---|---|
State | Accepted |
Commit | 73b404314372b36c788b709bcf18793f0a1abcbc |
Headers |
Return-Path: <development-bounces@lists.ipfire.org> Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384 client-signature ECDSA (P-384) client-digest SHA384) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by web04.haj.ipfire.org (Postfix) with ESMTPS id 4MCkSL5k9jz3wdF for <patchwork@web04.haj.ipfire.org>; Thu, 25 Aug 2022 00:32:18 +0000 (UTC) Received: from mail02.haj.ipfire.org (mail02.haj.ipfire.org [172.28.1.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384 client-signature ECDSA (P-384) client-digest SHA384) (Client CN "mail02.haj.ipfire.org", Issuer "R3" (verified OK)) by mail01.ipfire.org (Postfix) with ESMTPS id 4MCkSJ425dz46P; Thu, 25 Aug 2022 00:32:16 +0000 (UTC) Received: from mail02.haj.ipfire.org (localhost [127.0.0.1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4MCkSJ240zz2yjy; Thu, 25 Aug 2022 00:32:16 +0000 (UTC) Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384 client-signature ECDSA (P-384) client-digest SHA384) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4MCkSH0LQJz2y0x for <development@lists.ipfire.org>; Thu, 25 Aug 2022 00:32:15 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by mail01.ipfire.org (Postfix) with ESMTPSA id 4MCkSG0MMdzhH; Thu, 25 Aug 2022 00:32:13 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003ed25519; t=1661387534; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=3T0HBb1+j42CLhcLLPqcVCyFRfFz7kzlOgA1LzhWaVU=; b=rAINqU5L6twEv+GLjjCmYL1FrLUiE5dsICdSMa7ITB58JmuBlWXd9yQXebbAi/eh0VZrrC qdx0PFHMfm36ctDg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003rsa; t=1661387534; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=3T0HBb1+j42CLhcLLPqcVCyFRfFz7kzlOgA1LzhWaVU=; b=LdgLQTUmZwElAv2YYheK7GE15V51UdndLv4YVRGjAn9+cPS/RJrkRhREpjlsKua7wacood cyoaeZtsKiBJFVgj9Mkn8938jZXWpt9+6SvyEMhgWU0ObG1ya39JjJ44BUh+LvvcHhQnuZ U+dh1q1r6lghpy1eHqNWvHQlkUjAT+ulsiQXBGdjEpEWKha4bTI6IQeFuiX9hMsf1izrEj eZvwyRi+0cBhnsSCfHLUyJl+8Ysco0h4yVUKF2TCPbNkFfv+dGzco8bYiMwNO5VJvHnrjN 42x+1Xzytosg2LmC3xnj4Dt7Jxt+YxKU70JMeRyspS5698fC8y8SbYZ4Cr+v4Q== From: Jon Murphy <jon.murphy@ipfire.org> To: development@lists.ipfire.org Subject: [PATCH] crontab: add periodic cleanup the collectd RRD (graphs) Date: Wed, 24 Aug 2022 19:31:29 -0500 Message-Id: <20220825003129.3552746-1-jon.murphy@ipfire.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: development@lists.ipfire.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: IPFire development talk <development.lists.ipfire.org> List-Unsubscribe: <https://lists.ipfire.org/mailman/options/development>, <mailto:development-request@lists.ipfire.org?subject=unsubscribe> List-Archive: <http://lists.ipfire.org/pipermail/development/> List-Post: <mailto:development@lists.ipfire.org> List-Help: <mailto:development-request@lists.ipfire.org?subject=help> List-Subscribe: <https://lists.ipfire.org/mailman/listinfo/development>, <mailto:development-request@lists.ipfire.org?subject=subscribe> Errors-To: development-bounces@lists.ipfire.org Sender: "Development" <development-bounces@lists.ipfire.org> |
Series |
crontab: add periodic cleanup the collectd RRD (graphs)
|
|
Commit Message
jon
Aug. 25, 2022, 12:31 a.m. UTC
- Created (mostly) for old openvpn graphs
- RRD removed when no graph modification for +365 days
- chosen since graph max out is 365 days
- fcron job runs once per week
- chosen since this is just a cleanup and it doesnt need to run everyday
Note: logging can be added if needed.
Signed-off-by: Jon Murphy <jon.murphy@ipfire.org>
---
config/cron/crontab | 3 +++
1 file changed, 3 insertions(+)
Comments
Reviewed-by: Bernhard Bitsch <bbitsch@ipfire.org> Am 25.08.2022 um 02:31 schrieb Jon Murphy: > - Created (mostly) for old openvpn graphs > - RRD removed when no graph modification for +365 days > - chosen since graph max out is 365 days > - fcron job runs once per week > - chosen since this is just a cleanup and it doesnt need to run everyday > Note: logging can be added if needed. > > Signed-off-by: Jon Murphy <jon.murphy@ipfire.org> > --- > config/cron/crontab | 3 +++ > 1 file changed, 3 insertions(+) > > diff --git a/config/cron/crontab b/config/cron/crontab > index b727531fc..7088e0b75 100644 > --- a/config/cron/crontab > +++ b/config/cron/crontab > @@ -80,5 +80,8 @@ HOME=/ > # Cleanup the mail spool directory > %weekly * * /usr/sbin/dma-cleanup-spool > > +# Cleanup the collectd RRD (graphs) > +%weekly * * /bin/find /var/log/rrd -mtime +365 -type f -name '*.rrd' -delete -o -type d -empty -delete > + > # Update DNS trust anchor > %daily,random * * @runas(nobody) /usr/sbin/unbound-anchor -a /var/lib/unbound/root.key -c /etc/unbound/icannbundle.pem
Hello Jon, Thanks for the patch. Sorry if this sounds like a stupid question, but what is the rationale for this? -Michael > On 25 Aug 2022, at 01:31, Jon Murphy <jon.murphy@ipfire.org> wrote: > > - Created (mostly) for old openvpn graphs > - RRD removed when no graph modification for +365 days > - chosen since graph max out is 365 days > - fcron job runs once per week > - chosen since this is just a cleanup and it doesnt need to run everyday > Note: logging can be added if needed. > > Signed-off-by: Jon Murphy <jon.murphy@ipfire.org> > --- > config/cron/crontab | 3 +++ > 1 file changed, 3 insertions(+) > > diff --git a/config/cron/crontab b/config/cron/crontab > index b727531fc..7088e0b75 100644 > --- a/config/cron/crontab > +++ b/config/cron/crontab > @@ -80,5 +80,8 @@ HOME=/ > # Cleanup the mail spool directory > %weekly * * /usr/sbin/dma-cleanup-spool > > +# Cleanup the collectd RRD (graphs) > +%weekly * * /bin/find /var/log/rrd -mtime +365 -type f -name '*.rrd' -delete -o -type d -empty -delete > + > # Update DNS trust anchor > %daily,random * * @runas(nobody) /usr/sbin/unbound-anchor -a /var/lib/unbound/root.key -c /etc/unbound/icannbundle.pem > -- > 2.30.2 >
Hey Michael, Over the years I’ve used four different IPFire boxes. And during a "late-summer cleaning" I found lots of old RRD files. I ran this command: `find /var/log/rrd -mtime +365 -type f -name '*.rrd' -ls` And among those were lots of old unused openvpn graphs. I think I had 8-10 old openvpn graphs that were more than 1 year old. I had started to write a wiki page for users about finding and deleting old `/var/log/rrd/collectd/localhost/openvpn-*` RRDs. And then the wiki grew to ALL old graphs. Below is a snipped version of my original `find`. I think there were ~40 total old obsolete RRD files. ``` [root@ipfire ~] # find /var/log/rrd -mtime +365 -type f -name '*.rrd' -ls 1838 304 -rw-r--r-- 1 root root 307712 Nov 17 2019 /var/log/rrd/collectd/localhost/interface/if_octets-dummy0.rrd 1794 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-soc_dts0-virtual-0/temperature-temp1.rrd 1821 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-it8721-isa-0a30/temperature-temp2.rrd . . . 1831 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-it8721-isa-0a30/fanspeed-fan3.rrd 1888 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-soc_dts1-virtual-0/temperature-temp1.rrd 1785 152 -rw-r--r-- 1 root root 154888 Aug 12 2017 /var/log/rrd/collectd/localhost/sensors-smsc47b397-isa-0480/temperature-temp2.rrd . . . 1791 152 -rw-r--r-- 1 root root 154888 Aug 12 2017 /var/log/rrd/collectd/localhost/sensors-smsc47b397-isa-0480/fanspeed-fan3.rrd 1880 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/thermal-thermal_zone1/temperature-temperature.rrd 1875 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-coretemp-isa-0000/temperature-temp4.rrd 1874 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-coretemp-isa-0000/temperature-temp5.rrd 1781 152 -rw-r--r-- 1 root root 154888 Jan 13 2020 /var/log/rrd/collectd/localhost/thermal-thermal_zone2/temperature-temperature.rrd 1937 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /var/log/rrd/iptraffic/192.168.60.105.rrd 1938 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /var/log/rrd/iptraffic/192.168.60.147.rrd 1935 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /var/log/rrd/iptraffic/192.168.60.194.rrd [root@ipfire ~] # ``` As you can see some of the files were left over from 2017 and they did not get cleaned out during hardware changes. I obviously did not follow the migrations wiki! The wiki page was to be instructions for users to `find` the files and then to delete the files via the command line. While is was writing the wiki I remembered a Dev Mailing list message about sending users to the shell partially defeats the purpose of IPFire (bad paraphrase!). So instead of writing a wiki I wrote it a weekly fcronjob to the help everyone. More than you ever wanted to know! Hope this helps! Jon > On Aug 25, 2022, at 5:48 AM, Michael Tremer <michael.tremer@ipfire.org> wrote: > > Hello Jon, > > Thanks for the patch. > > Sorry if this sounds like a stupid question, but what is the rationale for this? > > -Michael > >> On 25 Aug 2022, at 01:31, Jon Murphy <jon.murphy@ipfire.org> wrote: >> >> - Created (mostly) for old openvpn graphs >> - RRD removed when no graph modification for +365 days >> - chosen since graph max out is 365 days >> - fcron job runs once per week >> - chosen since this is just a cleanup and it doesnt need to run everyday >> Note: logging can be added if needed. >> >> Signed-off-by: Jon Murphy <jon.murphy@ipfire.org> >> --- >> config/cron/crontab | 3 +++ >> 1 file changed, 3 insertions(+) >> >> diff --git a/config/cron/crontab b/config/cron/crontab >> index b727531fc..7088e0b75 100644 >> --- a/config/cron/crontab >> +++ b/config/cron/crontab >> @@ -80,5 +80,8 @@ HOME=/ >> # Cleanup the mail spool directory >> %weekly * * /usr/sbin/dma-cleanup-spool >> >> +# Cleanup the collectd RRD (graphs) >> +%weekly * * /bin/find /var/log/rrd -mtime +365 -type f -name '*.rrd' -delete -o -type d -empty -delete >> + >> # Update DNS trust anchor >> %daily,random * * @runas(nobody) /usr/sbin/unbound-anchor -a /var/lib/unbound/root.key -c /etc/unbound/icannbundle.pem >> -- >> 2.30.2 >> >
Hello Jon, > On 25 Aug 2022, at 15:40, Jon Murphy <jcmurphy26@gmail.com> wrote: > > Hey Michael, > > Over the years I’ve used four different IPFire boxes. And during a "late-summer cleaning" I found lots of old RRD files. I ran this command: > `find /var/log/rrd -mtime +365 -type f -name '*.rrd' -ls` Was this just because you had a look around in the file system, or did you suffer from low disk space? > And among those were lots of old unused openvpn graphs. I think I had 8-10 old openvpn graphs that were more than 1 year old. > > I had started to write a wiki page for users about finding and deleting old `/var/log/rrd/collectd/localhost/openvpn-*` RRDs. And then the wiki grew to ALL old graphs. > > Below is a snipped version of my original `find`. I think there were ~40 total old obsolete RRD files. > > ``` > [root@ipfire ~] # find /var/log/rrd -mtime +365 -type f -name '*.rrd' -ls > 1838 304 -rw-r--r-- 1 root root 307712 Nov 17 2019 /var/log/rrd/collectd/localhost/interface/if_octets-dummy0.rrd > 1794 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-soc_dts0-virtual-0/temperature-temp1.rrd > 1821 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-it8721-isa-0a30/temperature-temp2.rrd > . . . > 1831 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-it8721-isa-0a30/fanspeed-fan3.rrd > 1888 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-soc_dts1-virtual-0/temperature-temp1.rrd > 1785 152 -rw-r--r-- 1 root root 154888 Aug 12 2017 /var/log/rrd/collectd/localhost/sensors-smsc47b397-isa-0480/temperature-temp2.rrd > . . . > 1791 152 -rw-r--r-- 1 root root 154888 Aug 12 2017 /var/log/rrd/collectd/localhost/sensors-smsc47b397-isa-0480/fanspeed-fan3.rrd > 1880 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/thermal-thermal_zone1/temperature-temperature.rrd > 1875 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-coretemp-isa-0000/temperature-temp4.rrd > 1874 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-coretemp-isa-0000/temperature-temp5.rrd > 1781 152 -rw-r--r-- 1 root root 154888 Jan 13 2020 /var/log/rrd/collectd/localhost/thermal-thermal_zone2/temperature-temperature.rrd > 1937 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /var/log/rrd/iptraffic/192.168.60.105.rrd > 1938 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /var/log/rrd/iptraffic/192.168.60.147.rrd > 1935 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /var/log/rrd/iptraffic/192.168.60.194.rrd > [root@ipfire ~] # > ``` > > As you can see some of the files were left over from 2017 and they did not get cleaned out during hardware changes. I obviously did not follow the migrations wiki! Well, I suppose if it doesn’t happen automatically, we cannot rely on this :) I never follow this either. > The wiki page was to be instructions for users to `find` the files and then to delete the files via the command line. While is was writing the wiki I remembered a Dev Mailing list message about sending users to the shell partially defeats the purpose of IPFire (bad paraphrase!). So instead of writing a wiki I wrote it a weekly fcronjob to the help everyone. > More than you ever wanted to know! Hope this helps! Yes it does, but in the end my concern remains: Is it not possible that an OpenVPN connection is only being used once a year or something like that? Would we then not wipe any data? Wouldn’t those databases come back again? I suppose a database that has not been touched in over a year would not be carrying any information because that would have been rotated out of it (because that is how RRD databases work). So maybe that can go. But I would argue only then. So we would have to check what the longest time is those databases retain any information. Personally, I would redesign this whole thing a little. I would make them store a lot more data and I would even look for more metrics to collect. So, the extra space for a RRD database wouldn’t really bother me, even if the whole directory is going to use up gigabytes. If that is a problem for other users, I would like to know. Best, -Michael > Jon > > >> On Aug 25, 2022, at 5:48 AM, Michael Tremer <michael.tremer@ipfire.org> wrote: >> >> Hello Jon, >> >> Thanks for the patch. >> >> Sorry if this sounds like a stupid question, but what is the rationale for this? >> >> -Michael >> >>> On 25 Aug 2022, at 01:31, Jon Murphy <jon.murphy@ipfire.org> wrote: >>> >>> - Created (mostly) for old openvpn graphs >>> - RRD removed when no graph modification for +365 days >>> - chosen since graph max out is 365 days >>> - fcron job runs once per week >>> - chosen since this is just a cleanup and it doesnt need to run everyday >>> Note: logging can be added if needed. >>> >>> Signed-off-by: Jon Murphy <jon.murphy@ipfire.org> >>> --- >>> config/cron/crontab | 3 +++ >>> 1 file changed, 3 insertions(+) >>> >>> diff --git a/config/cron/crontab b/config/cron/crontab >>> index b727531fc..7088e0b75 100644 >>> --- a/config/cron/crontab >>> +++ b/config/cron/crontab >>> @@ -80,5 +80,8 @@ HOME=/ >>> # Cleanup the mail spool directory >>> %weekly * * /usr/sbin/dma-cleanup-spool >>> >>> +# Cleanup the collectd RRD (graphs) >>> +%weekly * * /bin/find /var/log/rrd -mtime +365 -type f -name '*.rrd' -delete -o -type d -empty -delete >>> + >>> # Update DNS trust anchor >>> %daily,random * * @runas(nobody) /usr/sbin/unbound-anchor -a /var/lib/unbound/root.key -c /etc/unbound/icannbundle.pem >>> -- >>> 2.30.2 >>> >> >
Michael, > On Aug 25, 2022, at 10:44 AM, Michael Tremer <michael.tremer@ipfire.org> wrote: > > Hello Jon, > >> On 25 Aug 2022, at 15:40, Jon Murphy <jcmurphy26@gmail.com <mailto:jcmurphy26@gmail.com>> wrote: >> >> Hey Michael, >> >> Over the years I’ve used four different IPFire boxes. And during a "late-summer cleaning" I found lots of old RRD files. I ran this command: >> `find /var/log/rrd -mtime +365 -type f -name '*.rrd' -ls` > > Was this just because you had a look around in the file system, or did you suffer from low disk space? Just looking… > >> And among those were lots of old unused openvpn graphs. I think I had 8-10 old openvpn graphs that were more than 1 year old. >> >> I had started to write a wiki page for users about finding and deleting old `/var/log/rrd/collectd/localhost/openvpn-*` RRDs. And then the wiki grew to ALL old graphs. >> >> Below is a snipped version of my original `find`. I think there were ~40 total old obsolete RRD files. >> >> ``` >> [root@ipfire ~] # find /var/log/rrd -mtime +365 -type f -name '*.rrd' -ls >> 1838 304 -rw-r--r-- 1 root root 307712 Nov 17 2019 /var/log/rrd/collectd/localhost/interface/if_octets-dummy0.rrd >> 1794 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-soc_dts0-virtual-0/temperature-temp1.rrd >> 1821 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-it8721-isa-0a30/temperature-temp2.rrd >> . . . >> 1831 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-it8721-isa-0a30/fanspeed-fan3.rrd >> 1888 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-soc_dts1-virtual-0/temperature-temp1.rrd >> 1785 152 -rw-r--r-- 1 root root 154888 Aug 12 2017 /var/log/rrd/collectd/localhost/sensors-smsc47b397-isa-0480/temperature-temp2.rrd >> . . . >> 1791 152 -rw-r--r-- 1 root root 154888 Aug 12 2017 /var/log/rrd/collectd/localhost/sensors-smsc47b397-isa-0480/fanspeed-fan3.rrd >> 1880 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/thermal-thermal_zone1/temperature-temperature.rrd >> 1875 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-coretemp-isa-0000/temperature-temp4.rrd >> 1874 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-coretemp-isa-0000/temperature-temp5.rrd >> 1781 152 -rw-r--r-- 1 root root 154888 Jan 13 2020 /var/log/rrd/collectd/localhost/thermal-thermal_zone2/temperature-temperature.rrd >> 1937 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /var/log/rrd/iptraffic/192.168.60.105.rrd >> 1938 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /var/log/rrd/iptraffic/192.168.60.147.rrd >> 1935 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /var/log/rrd/iptraffic/192.168.60.194.rrd >> [root@ipfire ~] # >> ``` >> >> As you can see some of the files were left over from 2017 and they did not get cleaned out during hardware changes. I obviously did not follow the migrations wiki! > > Well, I suppose if it doesn’t happen automatically, we cannot rely on this :) I never follow this either. > >> The wiki page was to be instructions for users to `find` the files and then to delete the files via the command line. While is was writing the wiki I remembered a Dev Mailing list message about sending users to the shell partially defeats the purpose of IPFire (bad paraphrase!). So instead of writing a wiki I wrote it a weekly fcronjob to the help everyone. > >> More than you ever wanted to know! Hope this helps! > > Yes it does, but in the end my concern remains: > > Is it not possible that an OpenVPN connection is only being used once a year or something like that? Would we then not wipe any data? Wouldn’t those databases come back again? I will test by wiping the current data and then connecting via OpenVPN to see if the databases and graphs recover. > > I suppose a database that has not been touched in over a year would not be carrying any information because that would have been rotated out of it (because that is how RRD databases work). So maybe that can go. But I would argue only then. So we would have to check what the longest time is those databases retain any information. > Who is the RRD guru for IPFire? I think there is an RRD dump command. I will research. > Personally, I would redesign this whole thing a little. I would make them store a lot more data and I would even look for more metrics to collect. So, the extra space for a RRD database wouldn’t really bother me, even if the whole directory is going to use up gigabytes. If that is a problem for other users, I would like to know. OK on my side! I am at 48% (Use%) for a 16 GB drive. I’d like to see a periodic bandwidth test (speedtest). Once in awhile I drop to near 0 Mbps bandwidth. Then it is time to restart the cable gateway and maybe the local network. Jon > > Best, > -Michael > >> Jon >> >> >>> On Aug 25, 2022, at 5:48 AM, Michael Tremer <michael.tremer@ipfire.org> wrote: >>> >>> Hello Jon, >>> >>> Thanks for the patch. >>> >>> Sorry if this sounds like a stupid question, but what is the rationale for this? >>> >>> -Michael >>> >>>> On 25 Aug 2022, at 01:31, Jon Murphy <jon.murphy@ipfire.org> wrote: >>>> >>>> - Created (mostly) for old openvpn graphs >>>> - RRD removed when no graph modification for +365 days >>>> - chosen since graph max out is 365 days >>>> - fcron job runs once per week >>>> - chosen since this is just a cleanup and it doesnt need to run everyday >>>> Note: logging can be added if needed. >>>> >>>> Signed-off-by: Jon Murphy <jon.murphy@ipfire.org> >>>> --- >>>> config/cron/crontab | 3 +++ >>>> 1 file changed, 3 insertions(+) >>>> >>>> diff --git a/config/cron/crontab b/config/cron/crontab >>>> index b727531fc..7088e0b75 100644 >>>> --- a/config/cron/crontab >>>> +++ b/config/cron/crontab >>>> @@ -80,5 +80,8 @@ HOME=/ >>>> # Cleanup the mail spool directory >>>> %weekly * * /usr/sbin/dma-cleanup-spool >>>> >>>> +# Cleanup the collectd RRD (graphs) >>>> +%weekly * * /bin/find /var/log/rrd -mtime +365 -type f -name '*.rrd' -delete -o -type d -empty -delete >>>> + >>>> # Update DNS trust anchor >>>> %daily,random * * @runas(nobody) /usr/sbin/unbound-anchor -a /var/lib/unbound/root.key -c /etc/unbound/icannbundle.pem >>>> -- >>>> 2.30.2
Hello, > On 25 Aug 2022, at 17:17, Jon Murphy <jcmurphy26@gmail.com> wrote: > > Michael, > > >> On Aug 25, 2022, at 10:44 AM, Michael Tremer <michael.tremer@ipfire.org> wrote: >> >> Hello Jon, >> >>> On 25 Aug 2022, at 15:40, Jon Murphy <jcmurphy26@gmail.com> wrote: >>> >>> Hey Michael, >>> >>> Over the years I’ve used four different IPFire boxes. And during a "late-summer cleaning" I found lots of old RRD files. I ran this command: >>> `find /var/log/rrd -mtime +365 -type f -name '*.rrd' -ls` >> >> Was this just because you had a look around in the file system, or did you suffer from low disk space? > > Just looking… :) > >> >>> And among those were lots of old unused openvpn graphs. I think I had 8-10 old openvpn graphs that were more than 1 year old. >>> >>> I had started to write a wiki page for users about finding and deleting old `/var/log/rrd/collectd/localhost/openvpn-*` RRDs. And then the wiki grew to ALL old graphs. >>> >>> Below is a snipped version of my original `find`. I think there were ~40 total old obsolete RRD files. >>> >>> ``` >>> [root@ipfire ~] # find /var/log/rrd -mtime +365 -type f -name '*.rrd' -ls >>> 1838 304 -rw-r--r-- 1 root root 307712 Nov 17 2019 /var/log/rrd/collectd/localhost/interface/if_octets-dummy0.rrd >>> 1794 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-soc_dts0-virtual-0/temperature-temp1.rrd >>> 1821 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-it8721-isa-0a30/temperature-temp2.rrd >>> . . . >>> 1831 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-it8721-isa-0a30/fanspeed-fan3.rrd >>> 1888 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-soc_dts1-virtual-0/temperature-temp1.rrd >>> 1785 152 -rw-r--r-- 1 root root 154888 Aug 12 2017 /var/log/rrd/collectd/localhost/sensors-smsc47b397-isa-0480/temperature-temp2.rrd >>> . . . >>> 1791 152 -rw-r--r-- 1 root root 154888 Aug 12 2017 /var/log/rrd/collectd/localhost/sensors-smsc47b397-isa-0480/fanspeed-fan3.rrd >>> 1880 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/thermal-thermal_zone1/temperature-temperature.rrd >>> 1875 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-coretemp-isa-0000/temperature-temp4.rrd >>> 1874 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-coretemp-isa-0000/temperature-temp5.rrd >>> 1781 152 -rw-r--r-- 1 root root 154888 Jan 13 2020 /var/log/rrd/collectd/localhost/thermal-thermal_zone2/temperature-temperature.rrd >>> 1937 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /var/log/rrd/iptraffic/192.168.60.105.rrd >>> 1938 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /var/log/rrd/iptraffic/192.168.60.147.rrd >>> 1935 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /var/log/rrd/iptraffic/192.168.60.194.rrd >>> [root@ipfire ~] # >>> ``` >>> >>> As you can see some of the files were left over from 2017 and they did not get cleaned out during hardware changes. I obviously did not follow the migrations wiki! >> >> Well, I suppose if it doesn’t happen automatically, we cannot rely on this :) I never follow this either. >> >>> The wiki page was to be instructions for users to `find` the files and then to delete the files via the command line. While is was writing the wiki I remembered a Dev Mailing list message about sending users to the shell partially defeats the purpose of IPFire (bad paraphrase!). So instead of writing a wiki I wrote it a weekly fcronjob to the help everyone. >> >>> More than you ever wanted to know! Hope this helps! >> >> Yes it does, but in the end my concern remains: >> >> Is it not possible that an OpenVPN connection is only being used once a year or something like that? Would we then not wipe any data? Wouldn’t those databases come back again? > > I will test by wiping the current data and then connecting via OpenVPN to see if the databases and graphs recover. I am quite confident that should work just fine. >> >> I suppose a database that has not been touched in over a year would not be carrying any information because that would have been rotated out of it (because that is how RRD databases work). So maybe that can go. But I would argue only then. So we would have to check what the longest time is those databases retain any information. >> > > Who is the RRD guru for IPFire? > > I think there is an RRD dump command. I will research. rrdtool dump /var/log/rrd/collectd/localhost/entropy/entropy.rrd That will show you a looooong list with all the data points. >> Personally, I would redesign this whole thing a little. I would make them store a lot more data and I would even look for more metrics to collect. So, the extra space for a RRD database wouldn’t really bother me, even if the whole directory is going to use up gigabytes. If that is a problem for other users, I would like to know. > > OK on my side! I am at 48% (Use%) for a 16 GB drive. Yes, 16GB is perfectly fine. > I’d like to see a periodic bandwidth test (speedtest). Once in awhile I drop to near 0 Mbps bandwidth. Then it is time to restart the cable gateway and maybe the local network. Yeah, that would be really nice to have :) I am just not sure how much traffic all IPFire devices like that would create. Best, -Michael > > > Jon > >> >> Best, >> -Michael >> >>> Jon >>> >>> >>>> On Aug 25, 2022, at 5:48 AM, Michael Tremer <michael.tremer@ipfire.org> wrote: >>>> >>>> Hello Jon, >>>> >>>> Thanks for the patch. >>>> >>>> Sorry if this sounds like a stupid question, but what is the rationale for this? >>>> >>>> -Michael >>>> >>>>> On 25 Aug 2022, at 01:31, Jon Murphy <jon.murphy@ipfire.org> wrote: >>>>> >>>>> - Created (mostly) for old openvpn graphs >>>>> - RRD removed when no graph modification for +365 days >>>>> - chosen since graph max out is 365 days >>>>> - fcron job runs once per week >>>>> - chosen since this is just a cleanup and it doesnt need to run everyday >>>>> Note: logging can be added if needed. >>>>> >>>>> Signed-off-by: Jon Murphy <jon.murphy@ipfire.org> >>>>> --- >>>>> config/cron/crontab | 3 +++ >>>>> 1 file changed, 3 insertions(+) >>>>> >>>>> diff --git a/config/cron/crontab b/config/cron/crontab >>>>> index b727531fc..7088e0b75 100644 >>>>> --- a/config/cron/crontab >>>>> +++ b/config/cron/crontab >>>>> @@ -80,5 +80,8 @@ HOME=/ >>>>> # Cleanup the mail spool directory >>>>> %weekly * * /usr/sbin/dma-cleanup-spool >>>>> >>>>> +# Cleanup the collectd RRD (graphs) >>>>> +%weekly * * /bin/find /var/log/rrd -mtime +365 -type f -name '*.rrd' -delete -o -type d -empty -delete >>>>> + >>>>> # Update DNS trust anchor >>>>> %daily,random * * @runas(nobody) /usr/sbin/unbound-anchor -a /var/lib/unbound/root.key -c /etc/unbound/icannbundle.pem >>>>> -- >>>>> 2.30.2
Michael, > On Aug 25, 2022, at 11:32 AM, Michael Tremer <michael.tremer@ipfire.org> wrote: > > Hello, > >> On 25 Aug 2022, at 17:17, Jon Murphy <jcmurphy26@gmail.com> wrote: >> >> Michael, >> >> >>> On Aug 25, 2022, at 10:44 AM, Michael Tremer <michael.tremer@ipfire.org> wrote: >>> >>> Hello Jon, >>> >>>> On 25 Aug 2022, at 15:40, Jon Murphy <jcmurphy26@gmail.com> wrote: >>>> >>>> Hey Michael, >>>> >>>> Over the years I’ve used four different IPFire boxes. And during a "late-summer cleaning" I found lots of old RRD files. I ran this command: >>>> `find /var/log/rrd -mtime +365 -type f -name '*.rrd' -ls` >>> >>> Was this just because you had a look around in the file system, or did you suffer from low disk space? >> >> Just looking… > > :) > >> >>> >>>> And among those were lots of old unused openvpn graphs. I think I had 8-10 old openvpn graphs that were more than 1 year old. >>>> >>>> I had started to write a wiki page for users about finding and deleting old `/var/log/rrd/collectd/localhost/openvpn-*` RRDs. And then the wiki grew to ALL old graphs. >>>> >>>> Below is a snipped version of my original `find`. I think there were ~40 total old obsolete RRD files. >>>> >>>> ``` >>>> [root@ipfire ~] # find /var/log/rrd -mtime +365 -type f -name '*.rrd' -ls >>>> 1838 304 -rw-r--r-- 1 root root 307712 Nov 17 2019 /var/log/rrd/collectd/localhost/interface/if_octets-dummy0.rrd >>>> 1794 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-soc_dts0-virtual-0/temperature-temp1.rrd >>>> 1821 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-it8721-isa-0a30/temperature-temp2.rrd >>>> . . . >>>> 1831 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-it8721-isa-0a30/fanspeed-fan3.rrd >>>> 1888 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-soc_dts1-virtual-0/temperature-temp1.rrd >>>> 1785 152 -rw-r--r-- 1 root root 154888 Aug 12 2017 /var/log/rrd/collectd/localhost/sensors-smsc47b397-isa-0480/temperature-temp2.rrd >>>> . . . >>>> 1791 152 -rw-r--r-- 1 root root 154888 Aug 12 2017 /var/log/rrd/collectd/localhost/sensors-smsc47b397-isa-0480/fanspeed-fan3.rrd >>>> 1880 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/thermal-thermal_zone1/temperature-temperature.rrd >>>> 1875 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-coretemp-isa-0000/temperature-temp4.rrd >>>> 1874 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-coretemp-isa-0000/temperature-temp5.rrd >>>> 1781 152 -rw-r--r-- 1 root root 154888 Jan 13 2020 /var/log/rrd/collectd/localhost/thermal-thermal_zone2/temperature-temperature.rrd >>>> 1937 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /var/log/rrd/iptraffic/192.168.60.105.rrd >>>> 1938 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /var/log/rrd/iptraffic/192.168.60.147.rrd >>>> 1935 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /var/log/rrd/iptraffic/192.168.60.194.rrd >>>> [root@ipfire ~] # >>>> ``` >>>> >>>> As you can see some of the files were left over from 2017 and they did not get cleaned out during hardware changes. I obviously did not follow the migrations wiki! >>> >>> Well, I suppose if it doesn’t happen automatically, we cannot rely on this :) I never follow this either. >>> >>>> The wiki page was to be instructions for users to `find` the files and then to delete the files via the command line. While is was writing the wiki I remembered a Dev Mailing list message about sending users to the shell partially defeats the purpose of IPFire (bad paraphrase!). So instead of writing a wiki I wrote it a weekly fcronjob to the help everyone. >>> >>>> More than you ever wanted to know! Hope this helps! >>> >>> Yes it does, but in the end my concern remains: >>> >>> Is it not possible that an OpenVPN connection is only being used once a year or something like that? Would we then not wipe any data? Wouldn’t those databases come back again? >> >> I will test by wiping the current data and then connecting via OpenVPN to see if the databases and graphs recover. > > I am quite confident that should work just fine. > >>> >>> I suppose a database that has not been touched in over a year would not be carrying any information because that would have been rotated out of it (because that is how RRD databases work). So maybe that can go. But I would argue only then. So we would have to check what the longest time is those databases retain any information. >>> >> >> Who is the RRD guru for IPFire? >> >> I think there is an RRD dump command. I will research. > > rrdtool dump /var/log/rrd/collectd/localhost/entropy/entropy.rrd > > That will show you a looooong list with all the data points. Looks like it lasts a year! Beginning of a section of round robin: ``` </database> </rra> <rra> <cf>AVERAGE</cf> <pdp_per_row>878</pdp_per_row> <!-- 26340 seconds --> <params> <xff>1.0000000000e-01</xff> </params> <cdp_prep> <ds> <primary_value>2.5600000000e+02</primary_value> <secondary_value>2.5600000000e+02</secondary_value> <value>2.0326400000e+05</value> <unknown_datapoints>0</unknown_datapoints> </ds> </cdp_prep> <database> <!-- 2021-08-24 09:04:00 CDT / 1629813840 --> <row><v>3.3458068337e+03</v></row> <!-- 2021-08-24 16:23:00 CDT / 1629840180 --> <row><v>3.3946542141e+03</v></row> <!-- 2021-08-24 23:42:00 CDT / 1629866520 --> <row><v>3.3813018223e+03</v></row> <!-- 2021-08-25 07:01:00 CDT / 1629892860 --> <row><v>3.3501125285e+03</v></row> ``` End of the round robin: ``` <!-- 2022-08-23 09:10:00 CDT / 1661263800 --> <row><v>2.5600000000e+02</v></row> <!-- 2022-08-23 16:29:00 CDT / 1661290140 --> <row><v>2.5600000000e+02</v></row> <!-- 2022-08-23 23:48:00 CDT / 1661316480 --> <row><v>2.5600000000e+02</v></row> <!-- 2022-08-24 07:07:00 CDT / 1661342820 --> <row><v>2.5600000000e+02</v></row> <!-- 2022-08-24 14:26:00 CDT / 1661369160 --> <row><v>2.5600000000e+02</v></row> <!-- 2022-08-24 21:45:00 CDT / 1661395500 --> <row><v>2.5600000000e+02</v></row> <!-- 2022-08-25 05:04:00 CDT / 1661421840 --> <row><v>2.5600000000e+02</v></row> </database> </rra> <rra> <cf>MIN</cf> <pdp_per_row>878</pdp_per_row> <!-- 26340 seconds --> ``` > >>> Personally, I would redesign this whole thing a little. I would make them store a lot more data and I would even look for more metrics to collect. So, the extra space for a RRD database wouldn’t really bother me, even if the whole directory is going to use up gigabytes. If that is a problem for other users, I would like to know. >> >> OK on my side! I am at 48% (Use%) for a 16 GB drive. > > Yes, 16GB is perfectly fine. > >> I’d like to see a periodic bandwidth test (speedtest). Once in awhile I drop to near 0 Mbps bandwidth. Then it is time to restart the cable gateway and maybe the local network. > > Yeah, that would be really nice to have :) I am just not sure how much traffic all IPFire devices like that would create. > > Best, > -Michael > >> >> >> Jon >> >>> >>> Best, >>> -Michael >>> >>>> Jon >>>> >>>> >>>>> On Aug 25, 2022, at 5:48 AM, Michael Tremer <michael.tremer@ipfire.org> wrote: >>>>> >>>>> Hello Jon, >>>>> >>>>> Thanks for the patch. >>>>> >>>>> Sorry if this sounds like a stupid question, but what is the rationale for this? >>>>> >>>>> -Michael >>>>> >>>>>> On 25 Aug 2022, at 01:31, Jon Murphy <jon.murphy@ipfire.org> wrote: >>>>>> >>>>>> - Created (mostly) for old openvpn graphs >>>>>> - RRD removed when no graph modification for +365 days >>>>>> - chosen since graph max out is 365 days >>>>>> - fcron job runs once per week >>>>>> - chosen since this is just a cleanup and it doesnt need to run everyday >>>>>> Note: logging can be added if needed. >>>>>> >>>>>> Signed-off-by: Jon Murphy <jon.murphy@ipfire.org> >>>>>> --- >>>>>> config/cron/crontab | 3 +++ >>>>>> 1 file changed, 3 insertions(+) >>>>>> >>>>>> diff --git a/config/cron/crontab b/config/cron/crontab >>>>>> index b727531fc..7088e0b75 100644 >>>>>> --- a/config/cron/crontab >>>>>> +++ b/config/cron/crontab >>>>>> @@ -80,5 +80,8 @@ HOME=/ >>>>>> # Cleanup the mail spool directory >>>>>> %weekly * * /usr/sbin/dma-cleanup-spool >>>>>> >>>>>> +# Cleanup the collectd RRD (graphs) >>>>>> +%weekly * * /bin/find /var/log/rrd -mtime +365 -type f -name '*.rrd' -delete -o -type d -empty -delete >>>>>> + >>>>>> # Update DNS trust anchor >>>>>> %daily,random * * @runas(nobody) /usr/sbin/unbound-anchor -a /var/lib/unbound/root.key -c /etc/unbound/icannbundle.pem >>>>>> -- >>>>>> 2.30.2
Michael, Last item below… If OK please add your "Acked-by:" or "Reviewed-by:". Jon > On Aug 25, 2022, at 11:47 AM, Jon Murphy <jcmurphy26@gmail.com> wrote: > > Michael, > >> On Aug 25, 2022, at 11:32 AM, Michael Tremer <michael.tremer@ipfire.org <mailto:michael.tremer@ipfire.org>> wrote: >> >> Hello, >> >>> On 25 Aug 2022, at 17:17, Jon Murphy <jcmurphy26@gmail.com <mailto:jcmurphy26@gmail.com>> wrote: >>> >>> Michael, >>> >>> >>>> On Aug 25, 2022, at 10:44 AM, Michael Tremer <michael.tremer@ipfire.org <mailto:michael.tremer@ipfire.org>> wrote: >>>> >>>> Hello Jon, >>>> >>>>> On 25 Aug 2022, at 15:40, Jon Murphy <jcmurphy26@gmail.com <mailto:jcmurphy26@gmail.com>> wrote: >>>>> >>>>> Hey Michael, >>>>> >>>>> Over the years I’ve used four different IPFire boxes. And during a "late-summer cleaning" I found lots of old RRD files. I ran this command: >>>>> `find /var/log/rrd -mtime +365 -type f -name '*.rrd' -ls` >>>> >>>> Was this just because you had a look around in the file system, or did you suffer from low disk space? >>> >>> Just looking… >> >> :) >> >>> >>>> >>>>> And among those were lots of old unused openvpn graphs. I think I had 8-10 old openvpn graphs that were more than 1 year old. >>>>> >>>>> I had started to write a wiki page for users about finding and deleting old `/var/log/rrd/collectd/localhost/openvpn-*` RRDs. And then the wiki grew to ALL old graphs. >>>>> >>>>> Below is a snipped version of my original `find`. I think there were ~40 total old obsolete RRD files. >>>>> >>>>> ``` >>>>> [root@ipfire ~] # find /var/log/rrd -mtime +365 -type f -name '*.rrd' -ls >>>>> 1838 304 -rw-r--r-- 1 root root 307712 Nov 17 2019 /var/log/rrd/collectd/localhost/interface/if_octets-dummy0.rrd >>>>> 1794 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-soc_dts0-virtual-0/temperature-temp1.rrd >>>>> 1821 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-it8721-isa-0a30/temperature-temp2.rrd >>>>> . . . >>>>> 1831 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-it8721-isa-0a30/fanspeed-fan3.rrd >>>>> 1888 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-soc_dts1-virtual-0/temperature-temp1.rrd >>>>> 1785 152 -rw-r--r-- 1 root root 154888 Aug 12 2017 /var/log/rrd/collectd/localhost/sensors-smsc47b397-isa-0480/temperature-temp2.rrd >>>>> . . . >>>>> 1791 152 -rw-r--r-- 1 root root 154888 Aug 12 2017 /var/log/rrd/collectd/localhost/sensors-smsc47b397-isa-0480/fanspeed-fan3.rrd >>>>> 1880 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/thermal-thermal_zone1/temperature-temperature.rrd >>>>> 1875 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-coretemp-isa-0000/temperature-temp4.rrd >>>>> 1874 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-coretemp-isa-0000/temperature-temp5.rrd >>>>> 1781 152 -rw-r--r-- 1 root root 154888 Jan 13 2020 /var/log/rrd/collectd/localhost/thermal-thermal_zone2/temperature-temperature.rrd >>>>> 1937 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /var/log/rrd/iptraffic/192.168.60.105.rrd >>>>> 1938 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /var/log/rrd/iptraffic/192.168.60.147.rrd >>>>> 1935 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /var/log/rrd/iptraffic/192.168.60.194.rrd >>>>> [root@ipfire ~] # >>>>> ``` >>>>> >>>>> As you can see some of the files were left over from 2017 and they did not get cleaned out during hardware changes. I obviously did not follow the migrations wiki! >>>> >>>> Well, I suppose if it doesn’t happen automatically, we cannot rely on this :) I never follow this either. >>>> >>>>> The wiki page was to be instructions for users to `find` the files and then to delete the files via the command line. While is was writing the wiki I remembered a Dev Mailing list message about sending users to the shell partially defeats the purpose of IPFire (bad paraphrase!). So instead of writing a wiki I wrote it a weekly fcronjob to the help everyone. >>>> >>>>> More than you ever wanted to know! Hope this helps! >>>> >>>> Yes it does, but in the end my concern remains: >>>> >>>> Is it not possible that an OpenVPN connection is only being used once a year or something like that? Would we then not wipe any data? Wouldn’t those databases come back again? >>> >>> I will test by wiping the current data and then connecting via OpenVPN to see if the databases and graphs recover. >> >> I am quite confident that should work just fine. I did a quick test and the folder and RRD file are both recreated. Cool! >> >>>> >>>> I suppose a database that has not been touched in over a year would not be carrying any information because that would have been rotated out of it (because that is how RRD databases work). So maybe that can go. But I would argue only then. So we would have to check what the longest time is those databases retain any information. >>>> >>> >>> Who is the RRD guru for IPFire? >>> >>> I think there is an RRD dump command. I will research. >> >> rrdtool dump /var/log/rrd/collectd/localhost/entropy/entropy.rrd >> >> That will show you a looooong list with all the data points. > > Looks like it lasts a year! > > Beginning of a section of round robin: > > ``` > </database> > </rra> > <rra> > <cf>AVERAGE</cf> > <pdp_per_row>878</pdp_per_row> <!-- 26340 seconds --> > > <params> > <xff>1.0000000000e-01</xff> > </params> > <cdp_prep> > <ds> > <primary_value>2.5600000000e+02</primary_value> > <secondary_value>2.5600000000e+02</secondary_value> > <value>2.0326400000e+05</value> > <unknown_datapoints>0</unknown_datapoints> > </ds> > </cdp_prep> > <database> > <!-- 2021-08-24 09:04:00 CDT / 1629813840 --> <row><v>3.3458068337e+03</v></row> > <!-- 2021-08-24 16:23:00 CDT / 1629840180 --> <row><v>3.3946542141e+03</v></row> > <!-- 2021-08-24 23:42:00 CDT / 1629866520 --> <row><v>3.3813018223e+03</v></row> > <!-- 2021-08-25 07:01:00 CDT / 1629892860 --> <row><v>3.3501125285e+03</v></row> > > ``` > > End of the round robin: > > ``` > <!-- 2022-08-23 09:10:00 CDT / 1661263800 --> <row><v>2.5600000000e+02</v></row> > <!-- 2022-08-23 16:29:00 CDT / 1661290140 --> <row><v>2.5600000000e+02</v></row> > <!-- 2022-08-23 23:48:00 CDT / 1661316480 --> <row><v>2.5600000000e+02</v></row> > <!-- 2022-08-24 07:07:00 CDT / 1661342820 --> <row><v>2.5600000000e+02</v></row> > <!-- 2022-08-24 14:26:00 CDT / 1661369160 --> <row><v>2.5600000000e+02</v></row> > <!-- 2022-08-24 21:45:00 CDT / 1661395500 --> <row><v>2.5600000000e+02</v></row> > <!-- 2022-08-25 05:04:00 CDT / 1661421840 --> <row><v>2.5600000000e+02</v></row> > </database> > </rra> > <rra> > <cf>MIN</cf> > <pdp_per_row>878</pdp_per_row> <!-- 26340 seconds --> > > ``` > >> >>>> Personally, I would redesign this whole thing a little. I would make them store a lot more data and I would even look for more metrics to collect. So, the extra space for a RRD database wouldn’t really bother me, even if the whole directory is going to use up gigabytes. If that is a problem for other users, I would like to know. >>> >>> OK on my side! I am at 48% (Use%) for a 16 GB drive. >> >> Yes, 16GB is perfectly fine. >> >>> I’d like to see a periodic bandwidth test (speedtest). Once in awhile I drop to near 0 Mbps bandwidth. Then it is time to restart the cable gateway and maybe the local network. >> >> Yeah, that would be really nice to have :) I am just not sure how much traffic all IPFire devices like that would create. >> >> Best, >> -Michael >> >>> >>> >>> Jon >>> >>>> >>>> Best, >>>> -Michael >>>> >>>>> Jon >>>>> >>>>> >>>>>> On Aug 25, 2022, at 5:48 AM, Michael Tremer <michael.tremer@ipfire.org <mailto:michael.tremer@ipfire.org>> wrote: >>>>>> >>>>>> Hello Jon, >>>>>> >>>>>> Thanks for the patch. >>>>>> >>>>>> Sorry if this sounds like a stupid question, but what is the rationale for this? >>>>>> >>>>>> -Michael >>>>>> >>>>>>> On 25 Aug 2022, at 01:31, Jon Murphy <jon.murphy@ipfire.org <mailto:jon.murphy@ipfire.org>> wrote: >>>>>>> >>>>>>> - Created (mostly) for old openvpn graphs >>>>>>> - RRD removed when no graph modification for +365 days >>>>>>> - chosen since graph max out is 365 days >>>>>>> - fcron job runs once per week >>>>>>> - chosen since this is just a cleanup and it doesnt need to run everyday >>>>>>> Note: logging can be added if needed. >>>>>>> >>>>>>> Signed-off-by: Jon Murphy <jon.murphy@ipfire.org <mailto:jon.murphy@ipfire.org>> >>>>>>> --- >>>>>>> config/cron/crontab | 3 +++ >>>>>>> 1 file changed, 3 insertions(+) >>>>>>> >>>>>>> diff --git a/config/cron/crontab b/config/cron/crontab >>>>>>> index b727531fc..7088e0b75 100644 >>>>>>> --- a/config/cron/crontab >>>>>>> +++ b/config/cron/crontab >>>>>>> @@ -80,5 +80,8 @@ HOME=/ >>>>>>> # Cleanup the mail spool directory >>>>>>> %weekly * * /usr/sbin/dma-cleanup-spool >>>>>>> >>>>>>> +# Cleanup the collectd RRD (graphs) >>>>>>> +%weekly * * /bin/find /var/log/rrd -mtime +365 -type f -name '*.rrd' -delete -o -type d -empty -delete >>>>>>> + >>>>>>> # Update DNS trust anchor >>>>>>> %daily,random * * @runas(nobody) /usr/sbin/unbound-anchor -a /var/lib/unbound/root.key -c /etc/unbound/icannbundle.pem >>>>>>> -- >>>>>>> 2.30.2 >
Bump / nudge... Is this OK? Jon > On Aug 25, 2022, at 4:34 PM, Jon Murphy <jcmurphy26@gmail.com> wrote: > > Michael, > > Last item below… > > If OK please add your "Acked-by:" or "Reviewed-by:". > > Jon > >> On Aug 25, 2022, at 11:47 AM, Jon Murphy <jcmurphy26@gmail.com <mailto:jcmurphy26@gmail.com>> wrote: >> >> Michael, >> >>> On Aug 25, 2022, at 11:32 AM, Michael Tremer <michael.tremer@ipfire.org <mailto:michael.tremer@ipfire.org>> wrote: >>> >>> Hello, >>> >>>> On 25 Aug 2022, at 17:17, Jon Murphy <jcmurphy26@gmail.com <mailto:jcmurphy26@gmail.com>> wrote: >>>> >>>> Michael, >>>> >>>> >>>>> On Aug 25, 2022, at 10:44 AM, Michael Tremer <michael.tremer@ipfire.org <mailto:michael.tremer@ipfire.org>> wrote: >>>>> >>>>> Hello Jon, >>>>> >>>>>> On 25 Aug 2022, at 15:40, Jon Murphy <jcmurphy26@gmail.com <mailto:jcmurphy26@gmail.com>> wrote: >>>>>> >>>>>> Hey Michael, >>>>>> >>>>>> Over the years I’ve used four different IPFire boxes. And during a "late-summer cleaning" I found lots of old RRD files. I ran this command: >>>>>> `find /var/log/rrd -mtime +365 -type f -name '*.rrd' -ls` >>>>> >>>>> Was this just because you had a look around in the file system, or did you suffer from low disk space? >>>> >>>> Just looking… >>> >>> :) >>> >>>> >>>>> >>>>>> And among those were lots of old unused openvpn graphs. I think I had 8-10 old openvpn graphs that were more than 1 year old. >>>>>> >>>>>> I had started to write a wiki page for users about finding and deleting old `/var/log/rrd/collectd/localhost/openvpn-*` RRDs. And then the wiki grew to ALL old graphs. >>>>>> >>>>>> Below is a snipped version of my original `find`. I think there were ~40 total old obsolete RRD files. >>>>>> >>>>>> ``` >>>>>> [root@ipfire ~] # find /var/log/rrd -mtime +365 -type f -name '*.rrd' -ls >>>>>> 1838 304 -rw-r--r-- 1 root root 307712 Nov 17 2019 /var/log/rrd/collectd/localhost/interface/if_octets-dummy0.rrd >>>>>> 1794 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-soc_dts0-virtual-0/temperature-temp1.rrd >>>>>> 1821 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-it8721-isa-0a30/temperature-temp2.rrd >>>>>> . . . >>>>>> 1831 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-it8721-isa-0a30/fanspeed-fan3.rrd >>>>>> 1888 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-soc_dts1-virtual-0/temperature-temp1.rrd >>>>>> 1785 152 -rw-r--r-- 1 root root 154888 Aug 12 2017 /var/log/rrd/collectd/localhost/sensors-smsc47b397-isa-0480/temperature-temp2.rrd >>>>>> . . . >>>>>> 1791 152 -rw-r--r-- 1 root root 154888 Aug 12 2017 /var/log/rrd/collectd/localhost/sensors-smsc47b397-isa-0480/fanspeed-fan3.rrd >>>>>> 1880 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/thermal-thermal_zone1/temperature-temperature.rrd >>>>>> 1875 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-coretemp-isa-0000/temperature-temp4.rrd >>>>>> 1874 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-coretemp-isa-0000/temperature-temp5.rrd >>>>>> 1781 152 -rw-r--r-- 1 root root 154888 Jan 13 2020 /var/log/rrd/collectd/localhost/thermal-thermal_zone2/temperature-temperature.rrd >>>>>> 1937 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /var/log/rrd/iptraffic/192.168.60.105.rrd >>>>>> 1938 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /var/log/rrd/iptraffic/192.168.60.147.rrd >>>>>> 1935 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /var/log/rrd/iptraffic/192.168.60.194.rrd >>>>>> [root@ipfire ~] # >>>>>> ``` >>>>>> >>>>>> As you can see some of the files were left over from 2017 and they did not get cleaned out during hardware changes. I obviously did not follow the migrations wiki! >>>>> >>>>> Well, I suppose if it doesn’t happen automatically, we cannot rely on this :) I never follow this either. >>>>> >>>>>> The wiki page was to be instructions for users to `find` the files and then to delete the files via the command line. While is was writing the wiki I remembered a Dev Mailing list message about sending users to the shell partially defeats the purpose of IPFire (bad paraphrase!). So instead of writing a wiki I wrote it a weekly fcronjob to the help everyone. >>>>> >>>>>> More than you ever wanted to know! Hope this helps! >>>>> >>>>> Yes it does, but in the end my concern remains: >>>>> >>>>> Is it not possible that an OpenVPN connection is only being used once a year or something like that? Would we then not wipe any data? Wouldn’t those databases come back again? >>>> >>>> I will test by wiping the current data and then connecting via OpenVPN to see if the databases and graphs recover. >>> >>> I am quite confident that should work just fine. > > I did a quick test and the folder and RRD file are both recreated. > > Cool! > >>> >>>>> >>>>> I suppose a database that has not been touched in over a year would not be carrying any information because that would have been rotated out of it (because that is how RRD databases work). So maybe that can go. But I would argue only then. So we would have to check what the longest time is those databases retain any information. >>>>> >>>> >>>> Who is the RRD guru for IPFire? >>>> >>>> I think there is an RRD dump command. I will research. >>> >>> rrdtool dump /var/log/rrd/collectd/localhost/entropy/entropy.rrd >>> >>> That will show you a looooong list with all the data points. >> >> Looks like it lasts a year! >> >> Beginning of a section of round robin: >> >> ``` >> </database> >> </rra> >> <rra> >> <cf>AVERAGE</cf> >> <pdp_per_row>878</pdp_per_row> <!-- 26340 seconds --> >> >> <params> >> <xff>1.0000000000e-01</xff> >> </params> >> <cdp_prep> >> <ds> >> <primary_value>2.5600000000e+02</primary_value> >> <secondary_value>2.5600000000e+02</secondary_value> >> <value>2.0326400000e+05</value> >> <unknown_datapoints>0</unknown_datapoints> >> </ds> >> </cdp_prep> >> <database> >> <!-- 2021-08-24 09:04:00 CDT / 1629813840 --> <row><v>3.3458068337e+03</v></row> >> <!-- 2021-08-24 16:23:00 CDT / 1629840180 --> <row><v>3.3946542141e+03</v></row> >> <!-- 2021-08-24 23:42:00 CDT / 1629866520 --> <row><v>3.3813018223e+03</v></row> >> <!-- 2021-08-25 07:01:00 CDT / 1629892860 --> <row><v>3.3501125285e+03</v></row> >> >> ``` >> >> End of the round robin: >> >> ``` >> <!-- 2022-08-23 09:10:00 CDT / 1661263800 --> <row><v>2.5600000000e+02</v></row> >> <!-- 2022-08-23 16:29:00 CDT / 1661290140 --> <row><v>2.5600000000e+02</v></row> >> <!-- 2022-08-23 23:48:00 CDT / 1661316480 --> <row><v>2.5600000000e+02</v></row> >> <!-- 2022-08-24 07:07:00 CDT / 1661342820 --> <row><v>2.5600000000e+02</v></row> >> <!-- 2022-08-24 14:26:00 CDT / 1661369160 --> <row><v>2.5600000000e+02</v></row> >> <!-- 2022-08-24 21:45:00 CDT / 1661395500 --> <row><v>2.5600000000e+02</v></row> >> <!-- 2022-08-25 05:04:00 CDT / 1661421840 --> <row><v>2.5600000000e+02</v></row> >> </database> >> </rra> >> <rra> >> <cf>MIN</cf> >> <pdp_per_row>878</pdp_per_row> <!-- 26340 seconds --> >> >> ``` >> >>> >>>>> Personally, I would redesign this whole thing a little. I would make them store a lot more data and I would even look for more metrics to collect. So, the extra space for a RRD database wouldn’t really bother me, even if the whole directory is going to use up gigabytes. If that is a problem for other users, I would like to know. >>>> >>>> OK on my side! I am at 48% (Use%) for a 16 GB drive. >>> >>> Yes, 16GB is perfectly fine. >>> >>>> I’d like to see a periodic bandwidth test (speedtest). Once in awhile I drop to near 0 Mbps bandwidth. Then it is time to restart the cable gateway and maybe the local network. >>> >>> Yeah, that would be really nice to have :) I am just not sure how much traffic all IPFire devices like that would create. >>> >>> Best, >>> -Michael >>> >>>> >>>> >>>> Jon >>>> >>>>> >>>>> Best, >>>>> -Michael >>>>> >>>>>> Jon >>>>>> >>>>>> >>>>>>> On Aug 25, 2022, at 5:48 AM, Michael Tremer <michael.tremer@ipfire.org <mailto:michael.tremer@ipfire.org>> wrote: >>>>>>> >>>>>>> Hello Jon, >>>>>>> >>>>>>> Thanks for the patch. >>>>>>> >>>>>>> Sorry if this sounds like a stupid question, but what is the rationale for this? >>>>>>> >>>>>>> -Michael >>>>>>> >>>>>>>> On 25 Aug 2022, at 01:31, Jon Murphy <jon.murphy@ipfire.org <mailto:jon.murphy@ipfire.org>> wrote: >>>>>>>> >>>>>>>> - Created (mostly) for old openvpn graphs >>>>>>>> - RRD removed when no graph modification for +365 days >>>>>>>> - chosen since graph max out is 365 days >>>>>>>> - fcron job runs once per week >>>>>>>> - chosen since this is just a cleanup and it doesnt need to run everyday >>>>>>>> Note: logging can be added if needed. >>>>>>>> >>>>>>>> Signed-off-by: Jon Murphy <jon.murphy@ipfire.org <mailto:jon.murphy@ipfire.org>> >>>>>>>> --- >>>>>>>> config/cron/crontab | 3 +++ >>>>>>>> 1 file changed, 3 insertions(+) >>>>>>>> >>>>>>>> diff --git a/config/cron/crontab b/config/cron/crontab >>>>>>>> index b727531fc..7088e0b75 100644 >>>>>>>> --- a/config/cron/crontab >>>>>>>> +++ b/config/cron/crontab >>>>>>>> @@ -80,5 +80,8 @@ HOME=/ >>>>>>>> # Cleanup the mail spool directory >>>>>>>> %weekly * * /usr/sbin/dma-cleanup-spool >>>>>>>> >>>>>>>> +# Cleanup the collectd RRD (graphs) >>>>>>>> +%weekly * * /bin/find /var/log/rrd -mtime +365 -type f -name '*.rrd' -delete -o -type d -empty -delete >>>>>>>> + >>>>>>>> # Update DNS trust anchor >>>>>>>> %daily,random * * @runas(nobody) /usr/sbin/unbound-anchor -a /var/lib/unbound/root.key -c /etc/unbound/icannbundle.pem >>>>>>>> -- >>>>>>>> 2.30.2 >> >
I think this is fine for me: Reviewed-by: Michael Tremer <michael.tremer@ipfire.org> Thanks for the nudge. -Michael > On 20 Sep 2022, at 03:28, Jon Murphy <jcmurphy26@gmail.com> wrote: > > Bump / nudge... Is this OK? > > > Jon > > >> On Aug 25, 2022, at 4:34 PM, Jon Murphy <jcmurphy26@gmail.com> wrote: >> >> Michael, >> >> Last item below… >> >> If OK please add your "Acked-by:" or "Reviewed-by:". >> >> Jon >> >>> On Aug 25, 2022, at 11:47 AM, Jon Murphy <jcmurphy26@gmail.com> wrote: >>> >>> Michael, >>> >>>> On Aug 25, 2022, at 11:32 AM, Michael Tremer <michael.tremer@ipfire.org> wrote: >>>> >>>> Hello, >>>> >>>>> On 25 Aug 2022, at 17:17, Jon Murphy <jcmurphy26@gmail.com> wrote: >>>>> >>>>> Michael, >>>>> >>>>> >>>>>> On Aug 25, 2022, at 10:44 AM, Michael Tremer <michael.tremer@ipfire.org> wrote: >>>>>> >>>>>> Hello Jon, >>>>>> >>>>>>> On 25 Aug 2022, at 15:40, Jon Murphy <jcmurphy26@gmail.com> wrote: >>>>>>> >>>>>>> Hey Michael, >>>>>>> >>>>>>> Over the years I’ve used four different IPFire boxes. And during a "late-summer cleaning" I found lots of old RRD files. I ran this command: >>>>>>> `find /var/log/rrd -mtime +365 -type f -name '*.rrd' -ls` >>>>>> >>>>>> Was this just because you had a look around in the file system, or did you suffer from low disk space? >>>>> >>>>> Just looking… >>>> >>>> :) >>>> >>>>> >>>>>> >>>>>>> And among those were lots of old unused openvpn graphs. I think I had 8-10 old openvpn graphs that were more than 1 year old. >>>>>>> >>>>>>> I had started to write a wiki page for users about finding and deleting old `/var/log/rrd/collectd/localhost/openvpn-*` RRDs. And then the wiki grew to ALL old graphs. >>>>>>> >>>>>>> Below is a snipped version of my original `find`. I think there were ~40 total old obsolete RRD files. >>>>>>> >>>>>>> ``` >>>>>>> [root@ipfire ~] # find /var/log/rrd -mtime +365 -type f -name '*.rrd' -ls >>>>>>> 1838 304 -rw-r--r-- 1 root root 307712 Nov 17 2019 /var/log/rrd/collectd/localhost/interface/if_octets-dummy0.rrd >>>>>>> 1794 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-soc_dts0-virtual-0/temperature-temp1.rrd >>>>>>> 1821 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-it8721-isa-0a30/temperature-temp2.rrd >>>>>>> . . . >>>>>>> 1831 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-it8721-isa-0a30/fanspeed-fan3.rrd >>>>>>> 1888 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-soc_dts1-virtual-0/temperature-temp1.rrd >>>>>>> 1785 152 -rw-r--r-- 1 root root 154888 Aug 12 2017 /var/log/rrd/collectd/localhost/sensors-smsc47b397-isa-0480/temperature-temp2.rrd >>>>>>> . . . >>>>>>> 1791 152 -rw-r--r-- 1 root root 154888 Aug 12 2017 /var/log/rrd/collectd/localhost/sensors-smsc47b397-isa-0480/fanspeed-fan3.rrd >>>>>>> 1880 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/thermal-thermal_zone1/temperature-temperature.rrd >>>>>>> 1875 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-coretemp-isa-0000/temperature-temp4.rrd >>>>>>> 1874 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-coretemp-isa-0000/temperature-temp5.rrd >>>>>>> 1781 152 -rw-r--r-- 1 root root 154888 Jan 13 2020 /var/log/rrd/collectd/localhost/thermal-thermal_zone2/temperature-temperature.rrd >>>>>>> 1937 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /var/log/rrd/iptraffic/192.168.60.105.rrd >>>>>>> 1938 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /var/log/rrd/iptraffic/192.168.60.147.rrd >>>>>>> 1935 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /var/log/rrd/iptraffic/192.168.60.194.rrd >>>>>>> [root@ipfire ~] # >>>>>>> ``` >>>>>>> >>>>>>> As you can see some of the files were left over from 2017 and they did not get cleaned out during hardware changes. I obviously did not follow the migrations wiki! >>>>>> >>>>>> Well, I suppose if it doesn’t happen automatically, we cannot rely on this :) I never follow this either. >>>>>> >>>>>>> The wiki page was to be instructions for users to `find` the files and then to delete the files via the command line. While is was writing the wiki I remembered a Dev Mailing list message about sending users to the shell partially defeats the purpose of IPFire (bad paraphrase!). So instead of writing a wiki I wrote it a weekly fcronjob to the help everyone. >>>>>> >>>>>>> More than you ever wanted to know! Hope this helps! >>>>>> >>>>>> Yes it does, but in the end my concern remains: >>>>>> >>>>>> Is it not possible that an OpenVPN connection is only being used once a year or something like that? Would we then not wipe any data? Wouldn’t those databases come back again? >>>>> >>>>> I will test by wiping the current data and then connecting via OpenVPN to see if the databases and graphs recover. >>>> >>>> I am quite confident that should work just fine. >> >> I did a quick test and the folder and RRD file are both recreated. >> >> Cool! >> >>>> >>>>>> >>>>>> I suppose a database that has not been touched in over a year would not be carrying any information because that would have been rotated out of it (because that is how RRD databases work). So maybe that can go. But I would argue only then. So we would have to check what the longest time is those databases retain any information. >>>>>> >>>>> >>>>> Who is the RRD guru for IPFire? >>>>> >>>>> I think there is an RRD dump command. I will research. >>>> >>>> rrdtool dump /var/log/rrd/collectd/localhost/entropy/entropy.rrd >>>> >>>> That will show you a looooong list with all the data points. >>> >>> Looks like it lasts a year! >>> >>> Beginning of a section of round robin: >>> >>> ``` >>> </database> >>> </rra> >>> <rra> >>> <cf>AVERAGE</cf> >>> <pdp_per_row>878</pdp_per_row> <!-- 26340 seconds --> >>> >>> <params> >>> <xff>1.0000000000e-01</xff> >>> </params> >>> <cdp_prep> >>> <ds> >>> <primary_value>2.5600000000e+02</primary_value> >>> <secondary_value>2.5600000000e+02</secondary_value> >>> <value>2.0326400000e+05</value> >>> <unknown_datapoints>0</unknown_datapoints> >>> </ds> >>> </cdp_prep> >>> <database> >>> <!-- 2021-08-24 09:04:00 CDT / 1629813840 --> <row><v>3.3458068337e+03</v></row> >>> <!-- 2021-08-24 16:23:00 CDT / 1629840180 --> <row><v>3.3946542141e+03</v></row> >>> <!-- 2021-08-24 23:42:00 CDT / 1629866520 --> <row><v>3.3813018223e+03</v></row> >>> <!-- 2021-08-25 07:01:00 CDT / 1629892860 --> <row><v>3.3501125285e+03</v></row> >>> >>> ``` >>> >>> End of the round robin: >>> >>> ``` >>> <!-- 2022-08-23 09:10:00 CDT / 1661263800 --> <row><v>2.5600000000e+02</v></row> >>> <!-- 2022-08-23 16:29:00 CDT / 1661290140 --> <row><v>2.5600000000e+02</v></row> >>> <!-- 2022-08-23 23:48:00 CDT / 1661316480 --> <row><v>2.5600000000e+02</v></row> >>> <!-- 2022-08-24 07:07:00 CDT / 1661342820 --> <row><v>2.5600000000e+02</v></row> >>> <!-- 2022-08-24 14:26:00 CDT / 1661369160 --> <row><v>2.5600000000e+02</v></row> >>> <!-- 2022-08-24 21:45:00 CDT / 1661395500 --> <row><v>2.5600000000e+02</v></row> >>> <!-- 2022-08-25 05:04:00 CDT / 1661421840 --> <row><v>2.5600000000e+02</v></row> >>> </database> >>> </rra> >>> <rra> >>> <cf>MIN</cf> >>> <pdp_per_row>878</pdp_per_row> <!-- 26340 seconds --> >>> >>> ``` >>> >>>> >>>>>> Personally, I would redesign this whole thing a little. I would make them store a lot more data and I would even look for more metrics to collect. So, the extra space for a RRD database wouldn’t really bother me, even if the whole directory is going to use up gigabytes. If that is a problem for other users, I would like to know. >>>>> >>>>> OK on my side! I am at 48% (Use%) for a 16 GB drive. >>>> >>>> Yes, 16GB is perfectly fine. >>>> >>>>> I’d like to see a periodic bandwidth test (speedtest). Once in awhile I drop to near 0 Mbps bandwidth. Then it is time to restart the cable gateway and maybe the local network. >>>> >>>> Yeah, that would be really nice to have :) I am just not sure how much traffic all IPFire devices like that would create. >>>> >>>> Best, >>>> -Michael >>>> >>>>> >>>>> >>>>> Jon >>>>> >>>>>> >>>>>> Best, >>>>>> -Michael >>>>>> >>>>>>> Jon >>>>>>> >>>>>>> >>>>>>>> On Aug 25, 2022, at 5:48 AM, Michael Tremer <michael.tremer@ipfire.org> wrote: >>>>>>>> >>>>>>>> Hello Jon, >>>>>>>> >>>>>>>> Thanks for the patch. >>>>>>>> >>>>>>>> Sorry if this sounds like a stupid question, but what is the rationale for this? >>>>>>>> >>>>>>>> -Michael >>>>>>>> >>>>>>>>> On 25 Aug 2022, at 01:31, Jon Murphy <jon.murphy@ipfire.org> wrote: >>>>>>>>> >>>>>>>>> - Created (mostly) for old openvpn graphs >>>>>>>>> - RRD removed when no graph modification for +365 days >>>>>>>>> - chosen since graph max out is 365 days >>>>>>>>> - fcron job runs once per week >>>>>>>>> - chosen since this is just a cleanup and it doesnt need to run everyday >>>>>>>>> Note: logging can be added if needed. >>>>>>>>> >>>>>>>>> Signed-off-by: Jon Murphy <jon.murphy@ipfire.org> >>>>>>>>> --- >>>>>>>>> config/cron/crontab | 3 +++ >>>>>>>>> 1 file changed, 3 insertions(+) >>>>>>>>> >>>>>>>>> diff --git a/config/cron/crontab b/config/cron/crontab >>>>>>>>> index b727531fc..7088e0b75 100644 >>>>>>>>> --- a/config/cron/crontab >>>>>>>>> +++ b/config/cron/crontab >>>>>>>>> @@ -80,5 +80,8 @@ HOME=/ >>>>>>>>> # Cleanup the mail spool directory >>>>>>>>> %weekly * * /usr/sbin/dma-cleanup-spool >>>>>>>>> >>>>>>>>> +# Cleanup the collectd RRD (graphs) >>>>>>>>> +%weekly * * /bin/find /var/log/rrd -mtime +365 -type f -name '*.rrd' -delete -o -type d -empty -delete >>>>>>>>> + >>>>>>>>> # Update DNS trust anchor >>>>>>>>> %daily,random * * @runas(nobody) /usr/sbin/unbound-anchor -a /var/lib/unbound/root.key -c /etc/unbound/icannbundle.pem >>>>>>>>> -- >>>>>>>>> 2.30.2 >>> >> >
diff --git a/config/cron/crontab b/config/cron/crontab index b727531fc..7088e0b75 100644 --- a/config/cron/crontab +++ b/config/cron/crontab @@ -80,5 +80,8 @@ HOME=/ # Cleanup the mail spool directory %weekly * * /usr/sbin/dma-cleanup-spool +# Cleanup the collectd RRD (graphs) +%weekly * * /bin/find /var/log/rrd -mtime +365 -type f -name '*.rrd' -delete -o -type d -empty -delete + # Update DNS trust anchor %daily,random * * @runas(nobody) /usr/sbin/unbound-anchor -a /var/lib/unbound/root.key -c /etc/unbound/icannbundle.pem