Message ID | 20220309225655.4472-10-robin.roevens@disroot.org |
---|---|
State | Superseded |
Headers |
Return-Path: <development-bounces@lists.ipfire.org> Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by web04.haj.ipfire.org (Postfix) with ESMTPS id 4KDSJS0xvLz3xq8 for <patchwork@web04.haj.ipfire.org>; Wed, 9 Mar 2022 22:57:28 +0000 (UTC) Received: from mail02.haj.ipfire.org (mail02.haj.ipfire.org [172.28.1.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail02.haj.ipfire.org", Issuer "R3" (verified OK)) by mail01.ipfire.org (Postfix) with ESMTPS id 4KDSJM1vYqz2RJ; Wed, 9 Mar 2022 22:57:23 +0000 (UTC) Received: from mail02.haj.ipfire.org (localhost [127.0.0.1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4KDSJM0djNz2yj9; Wed, 9 Mar 2022 22:57:23 +0000 (UTC) Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384 client-signature ECDSA (P-384) client-digest SHA384) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4KDSJL1Cg0z2xWt for <development@lists.ipfire.org>; Wed, 9 Mar 2022 22:57:22 +0000 (UTC) Received: from knopi.disroot.org (knopi.disroot.org [178.21.23.139]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mail01.ipfire.org (Postfix) with ESMTPS id 4KDSJJ2Y0Pz1DM for <development@lists.ipfire.org>; Wed, 9 Mar 2022 22:57:20 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by disroot.org (Postfix) with ESMTP id 8DF40400A8 for <development@lists.ipfire.org>; Wed, 9 Mar 2022 23:57:19 +0100 (CET) X-Virus-Scanned: SPAM Filter at disroot.org Received: from knopi.disroot.org ([127.0.0.1]) by localhost (disroot.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Gf6ttnKVMqtA for <development@lists.ipfire.org>; Wed, 9 Mar 2022 23:57:18 +0100 (CET) Received: from chojin.sicho.home (amaterasu.sicho.home [192.168.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (no client certificate requested) (Authenticated sender) by hachiman (MailScanner Milter) with SMTP id BD76C1B98F; Wed, 9 Mar 2022 23:57:10 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=disroot.org; s=mail; t=1646866637; bh=EbFOVEGvcLkWqrFQSQj4F9DH2ya/HjA61c43PUXADDM=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=X5KtcYGHnqR4NMyNBJfVQQlkkd7y/tCmrRkDi/O2GflEjbzAftMZTxZLTeGLuJRwe y1wUdoS4jGinPOMcweiZ/L5/fypThK44ekhRuj25LDEn848yUITvAnp7nOiyJAZuQm wdYTi2MRMuAHnXR70tkzbmFMN6AFN6me1Fv9VElbWwwjosO6J5A69OQszBGwvCMx1v P3sXV8kgpCDwl0rRRD1aoRrDBG5LN5YJ7+qfZtiBwlF0/FV2Sl/Za7SI2Pax/LTSLZ XxCH5ONJYT/Ur1PMhMfbdrhnlh2Af83TV2WvyfLgdod+i/amen9JQtI1ZLnsQi8HjP SZGli1wpm07BA== From: Robin Roevens <robin.roevens@disroot.org> To: development@lists.ipfire.org Subject: [PATCH 9/9] pakfire: Redesign update output and workflow Date: Wed, 9 Mar 2022 23:56:55 +0100 Message-Id: <20220309225655.4472-10-robin.roevens@disroot.org> In-Reply-To: <20220309225655.4472-1-robin.roevens@disroot.org> References: <20220309225655.4472-1-robin.roevens@disroot.org> Mime-Version: 1.0 Content-Transfer-Encoding: 8bit X-sicho-MailScanner-ID: BD76C1B98F.A8A80 X-sicho-MailScanner: Found to be clean X-sicho-MailScanner-From: robin.roevens@disroot.org X-sicho-MailScanner-Watermark: 1647471431.46625@Z4GJohCXoiZQ57FQG6iTgQ ARC-Seal: i=1; s=202003rsa; d=lists.ipfire.org; t=1646866640; a=rsa-sha256; cv=none; b=exX84jp3maIAKbNltU2pnHMTUEO4lwqpeolXMLI63R+M1hXXs0Eq6+Ayn6ras/XOxMpwZT JdW64kapMtnXhsmHTJwEl2b083M2GG60ItDHho6CLyrDWAYu6lT8Uf0fTLorwe+z6Gb7Ot uZWui2rbkUO1U5DASqyo9VqLpj52e4iADO2sCpvsfKvnxPN9aLcGsNVUx13tAddByInZKo loh6nnkW9FJkW1Jr12hbEpJpjmmvhF0uzNH9mYrcKzlr+j6hyW5RIURZVuDGHaYuSxaknK w5Et/wqJBQL9vOKKT4eUzjgXttr/is0dRtkVzuTSVEEpJcsVBpfl9Z8JCsJdVA== ARC-Authentication-Results: i=1; mail01.ipfire.org; dkim=pass header.d=disroot.org header.s=mail header.b=X5KtcYGH; dmarc=pass (policy=quarantine) header.from=disroot.org; spf=pass (mail01.ipfire.org: domain of robin.roevens@disroot.org designates 178.21.23.139 as permitted sender) smtp.mailfrom=robin.roevens@disroot.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=lists.ipfire.org; s=202003rsa; t=1646866640; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jKRq010RhaJl/rgcLH6HZ3shcQ40uFSvt0HJAmIm0S4=; b=RR5X7XVU8Wgy9gHSBL05zCXGK0bL7G4eP8blEy6qEk4CbRpp8TPHMcO5OQpr5uEwOWqkU+ HtRAVrdNewvOIrgDKzeVRc0k4qxvyvRqmddd9b5atn/bFwc+r/r/ckD2wqMMCTEPuX8T/V WXsPTccRDULRHvxGAjqyxR1pqb0MRPf756Z/x/DDA2UWSjaTGu5Y0b8+LJpXk2uz6L7uAs 6YVdgKAdGtsLBGe4Sc5i5gR0JuW2Saqc20XawizWgjyO9olGPgoq/dFKJP5dbugGFsOWcx udPrztzFODNUaDuAJ1dg9gDPyYCAqQZMa1Q4RciRV7JEfxA6GM7J6s+yCSr13Q== Authentication-Results: mail01.ipfire.org; dkim=pass header.d=disroot.org header.s=mail header.b=X5KtcYGH; dmarc=pass (policy=quarantine) header.from=disroot.org; spf=pass (mail01.ipfire.org: domain of robin.roevens@disroot.org designates 178.21.23.139 as permitted sender) smtp.mailfrom=robin.roevens@disroot.org X-Rspamd-Server: mail01.haj.ipfire.org X-Spamd-Result: default: False [-4.82 / 11.00]; BAYES_HAM(-3.00)[99.99%]; IP_REPUTATION_HAM(-1.15)[asn: 50673(-0.33), country: NL(-0.01), ip: 178.21.23.139(-0.82)]; MID_CONTAINS_FROM(1.00)[]; NEURAL_HAM(-0.98)[-0.982]; SPF_REPUTATION_HAM(-0.68)[-0.67693222528735]; DMARC_POLICY_ALLOW(-0.50)[disroot.org,quarantine]; MV_CASE(0.50)[]; R_MISSING_CHARSET(0.50)[]; R_SPF_ALLOW(-0.20)[+a]; R_DKIM_ALLOW(-0.20)[disroot.org:s=mail]; MIME_GOOD(-0.10)[text/plain]; MX_GOOD(-0.01)[]; ASN(0.00)[asn:50673, ipnet:178.21.23.0/24, country:NL]; FROM_EQ_ENVFROM(0.00)[]; ARC_SIGNED(0.00)[lists.ipfire.org:s=202003rsa:i=1]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; PREVIOUSLY_DELIVERED(0.00)[development@lists.ipfire.org]; DKIM_TRACE(0.00)[disroot.org:+]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; TO_DN_SOME(0.00)[]; ARC_NA(0.00)[] X-Rspamd-Queue-Id: 4KDSJJ2Y0Pz1DM X-BeenThere: development@lists.ipfire.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: IPFire development talk <development.lists.ipfire.org> List-Unsubscribe: <https://lists.ipfire.org/mailman/options/development>, <mailto:development-request@lists.ipfire.org?subject=unsubscribe> List-Archive: <http://lists.ipfire.org/pipermail/development/> List-Post: <mailto:development@lists.ipfire.org> List-Help: <mailto:development-request@lists.ipfire.org?subject=help> List-Subscribe: <https://lists.ipfire.org/mailman/listinfo/development>, <mailto:development-request@lists.ipfire.org?subject=subscribe> Errors-To: development-bounces@lists.ipfire.org Sender: "Development" <development-bounces@lists.ipfire.org> |
Series |
pakfire: remove dup. code + seperate ui/logic
|
|
Commit Message
Robin Roevens
March 9, 2022, 10:56 p.m. UTC
- Revamped 'pakfire update' more in the style of 'pakfire
install' for a more consistent look and feel.
- Add core update details to output
- Add new dependencies to install to output
- First collect all upgrade info, then perform core update
together with available package updates and after user
confirmation (without -y).
Signed-off-by: Robin Roevens <robin.roevens@disroot.org>
---
src/pakfire/pakfire | 79 +++++++++++++++++++++++++++++----------------
1 file changed, 51 insertions(+), 28 deletions(-)
Comments
Hello, > On 9 Mar 2022, at 22:56, Robin Roevens <robin.roevens@disroot.org> wrote: > > - Revamped 'pakfire update' more in the style of 'pakfire > install' for a more consistent look and feel. > - Add core update details to output > - Add new dependencies to install to output > - First collect all upgrade info, then perform core update > together with available package updates and after user > confirmation (without -y). I am sorry to disappoint Tom, but this is quite likely a no-can-do. The problem is that we are repeatedly tight on disk space which makes downloading everything first a bit of a problem. We do not have any mechanism that checks whether enough space for either the downloads or for extracting the update is available, which is why the old way was more of a step-by-step system. I don’t like the old system at all, but I would propose to add a function that checks if enough disk space is available before it asks the user whether they want to continue. That should consider: * The download size of the updates * The download size of all add-ons * The extracted size (for simplicity probably assuming that the core update is only adding new files) * The extracted size of all add-ons Then, we can even drop some last checks in the update scripts that aborted the update process if disk space was running low. How does that sound? -Michael > > Signed-off-by: Robin Roevens <robin.roevens@disroot.org> > --- > src/pakfire/pakfire | 79 +++++++++++++++++++++++++++++---------------- > 1 file changed, 51 insertions(+), 28 deletions(-) > > diff --git a/src/pakfire/pakfire b/src/pakfire/pakfire > index 9935481a5..0a144c517 100644 > --- a/src/pakfire/pakfire > +++ b/src/pakfire/pakfire > @@ -258,51 +258,76 @@ > &Pakfire::getcoredb("$force"); > > } elsif ("$ARGV[0]" eq "upgrade") { > - my $use_color = ""; > - my $reset_color = ""; > - > - if ("$Pakfire::enable_colors" eq "1") { > - $reset_color = "$Pakfire::color{'normal'}"; > - $use_color = "$Pakfire::color{'lightpurple'}"; > - } > - > &Pakfire::message("CORE INFO: Checking for Core updates..."); > - > ### Make sure that the core db is not outdated. > &Pakfire::getcoredb("noforce"); > - my %coredb = &Pakfire::coredbinfo(); > > - if (defined $coredb{'AvailableRelease'}) { > - &Pakfire::upgradecore(); > - } else { > - &Pakfire::message("CORE WARN: No new Core upgrades available. You are on release ".$coredb{'Release'}); > - } > + my %coredb = &Pakfire::coredbinfo(); > + &Pakfire::message("CORE WARN: No new Core upgrades available. You are on release ".$coredb{'Release'}) unless (defined $coredb{'AvailableRelease'}); > > &Pakfire::message("PAKFIRE INFO: Checking for package updates..."); > ### Make sure that the package list is not outdated. > &Pakfire::dbgetlist("noforce"); > > my @deps = (); > - if (my %upgradepaks = &Pakfire::dblist("upgrade")) { > - # Resolve the dependencies of the to be upgraded packages > - @deps = &Pakfire::resolvedeps_recursive(keys %upgradepaks); > + my %upgradepaks = &Pakfire::dblist("upgrade"); > > - foreach $pak (sort keys %upgradepaks) { > - print "${use_color}Update: $pak\nVersion: $upgradepaks{$pak}{'ProgVersion'} -> $upgradepaks{$pak}{'AvailableProgVersion'}\n"; > - print "Release: $upgradepaks{$pak}{'Release'} -> $upgradepaks{$pak}{'AvailableRelease'}${reset_color}\n"; > - } > + # Resolve the dependencies of the to be upgraded packages > + @deps = &Pakfire::resolvedeps_recursive(keys %upgradepaks) if (%upgradepaks); > + &Pakfire::message("PAKFIRE WARN: No new package upgrades available.") unless (@deps); > + > + if (defined $coredb{'AvailableRelease'} || %upgradepaks) { > + &Pakfire::message(""); > &Pakfire::message(""); > - &Pakfire::message("PAKFIRE UPGR: We are going to install all packages listed above."); > + &Pakfire::message("PAKFIRE INFO: Upgrade summary:"); > + &Pakfire::message(""); > + > + if (defined $coredb{'AvailableRelease'}) { > + &Pakfire::message("CORE INFO: Core-Update $coredb{'CoreVersion'} to install:"); > + &Pakfire::message("CORE UPGR: Release: $coredb{'Release'} -> $coredb{'AvailableRelease'}"); > + &Pakfire::message(""); > + } > + > + if (@deps) { > + &Pakfire::message("PAKFIRE INFO: New dependencies to install:"); > + my $totalsize = 0; > + foreach $pak (@deps) { > + unless (defined $upgradepaks{$pak} || &Pakfire::isinstalled($pak) == 0) { > + my $size = &Pakfire::getsize("$pak"); > + $totalsize += $size; > + $size = &Pakfire::beautifysize($size); > + &Pakfire::message("PAKFIRE INFO: $pak \t - $size"); > + } > + } > + $totalsize = &Pakfire::beautifysize($totalsize); > + &Pakfire::message(""); > + &Pakfire::message("PAKFIRE INFO: Total size: \t ~ $totalsize"); > + &Pakfire::message(""); > + } > + > + if (%upgradepaks) { > + &Pakfire::message("PAKFIRE INFO: Packages to upgrade:"); > + foreach $pak (sort keys %upgradepaks) { > + &Pakfire::message("PAKFIRE UPGR: $pak\t$upgradepaks{$pak}{'ProgVersion'}-$upgradepaks{$pak}{'Release'} -> $upgradepaks{$pak}{'AvailableProgVersion'}-$upgradepaks{$pak}{'AvailableRelease'}"); > + } > + &Pakfire::message(""); > + } > + > if ($interactive) { > - &Pakfire::message("PAKFIRE INFO: Is this okay? [y/N]"); > + &Pakfire::message("PAKFIRE INFO: Is this okay? [y/N]"); > my $ret = <STDIN>; > chomp($ret); > &Pakfire::logger("PAKFIRE INFO: Answer: $ret"); > if ( $ret ne "y" ) { > - &Pakfire::message("PAKFIRE ERROR: Installation aborted."); > - exit 1; > + &Pakfire::message("PAKFIRE ERROR: Installation aborted."); > + exit 1; > } > } > + > + # Perform core update > + if (defined $coredb{'AvailableRelease'}) { > + &Pakfire::upgradecore(); > + } > > # Download packages > foreach $pak (sort keys %upgradepaks) { > @@ -323,8 +348,6 @@ > foreach $pak (sort keys %upgradepaks) { > &Pakfire::upgradepak("$pak"); > } > - } else { > - &Pakfire::message("PAKFIRE WARN: No new package upgrades available."); > } > > } elsif ("$ARGV[0]" eq "list") { > -- > 2.34.1 > > > -- > Dit bericht is gescanned op virussen en andere gevaarlijke > inhoud door MailScanner en lijkt schoon te zijn. >
Michael Tremer schreef op ma 21-03-2022 om 16:36 [+0000]: > Hello, > > > On 9 Mar 2022, at 22:56, Robin Roevens <robin.roevens@disroot.org> > > wrote: > > > > - Revamped 'pakfire update' more in the style of 'pakfire > > install' for a more consistent look and feel. > > - Add core update details to output > > - Add new dependencies to install to output > > - First collect all upgrade info, then perform core update > > together with available package updates and after user > > confirmation (without -y). > > I am sorry to disappoint Tom, but this is quite likely a no-can-do. > > The problem is that we are repeatedly tight on disk space which makes > downloading everything first a bit of a problem. I'm not sure what you mean? Originally the upgrade function forced a core upgrade first, then retrieved info about pak updates and requested confirmation to install them. I changed this into first retrieving all pak and core update info (not the paks or core themselves), presenting the info to the user and asking confirmation. After that the core is downloaded and installed, then the paks are downloaded and installed. So in terms of disk space usage, nothing should have changed except that pak metadata is now downloaded before the core update is performed. But those few text files should not be the problem I assume? > > We do not have any mechanism that checks whether enough space for > either the downloads or for extracting the update is available, which > is why the old way was more of a step-by-step system. > > I don’t like the old system at all, but I would propose to add a > function that checks if enough disk space is available before it asks > the user whether they want to continue. > > That should consider: > > * The download size of the updates > * The download size of all add-ons > * The extracted size (for simplicity probably assuming that the core > update is only adding new files) > * The extracted size of all add-ons Great minds think alike, I suppose :-) I was already in the middle of trying to implement size checking. But I didn't want the current patchset submission to be postponed for that. (The last patch was already some kind of 'extra' and not really that related to the purpose of that patchset (the cleanup of dblist and using extra metadata)) I already did want to check some things about the size calculation here on the mailing list, but I figured that this patchset should first need to be accepted, as the size calculation would depend on it. But now that we are on this topic.. I wanted to verify here if the size recorded in the meta-files, is the size of the tar-files? And since everything is tarred without compression (or so I think?) this should more or less be equal to the installed size ? Or am I mistaken? If we ever decide to actually compress those we will need to record uncompressed size in the metadata. Then for calculation, I was thinking about separating temporary space and installed space: * core update: -- Required tempspace: size of new core (tar-file) -- required installspace: size of new core * new dependencies: -- required tempspace: sum of size of all required deps (tar files) -- required installspace: sum of size of all required deps * pak updates: -- required tempspace: sum of size of all required paks (tar files) -- required installspace: for each pak: size of installed pak version - size of new pak version in the end total required space is tempspace + installspace, if installspace is positive. It can in theory also be negative (no core update and size of updated paks is less than previous versions), then required space = tempspace.. I'm not sure if downloaded pakfiles/core-updates are deleted after install..? Or after some time ? I did not immediately find it in the code. We could implement some cache pruning ('pakfire clear' or 'pakfire clear-cache' or so?) this size diff at least for pak updates should be quite accurate, as previous pak is uninstalled and new pak is installed during update. I have been trying to calculate the actual size of a core update.. but only just now I figured that a core update file is probably not a full ipfire fs, but only the diff in files against previous core update :-) For the actual size of a core update, we would need to have a diff of the size of a clean ipfire on previous core and a clean ipfire on next core.. But I don't think that is data we have already available or could easily have available ? So assuming core update only adds files is probably indeed the safest straight-forward method. Other method could be to assume a percentage of new files-size in a core update..? Robin > > Then, we can even drop some last checks in the update scripts that > aborted the update process if disk space was running low. > > How does that sound? > > -Michael > > > > > Signed-off-by: Robin Roevens <robin.roevens@disroot.org> > > --- > > src/pakfire/pakfire | 79 +++++++++++++++++++++++++++++------------- > > --- > > 1 file changed, 51 insertions(+), 28 deletions(-) > > > > diff --git a/src/pakfire/pakfire b/src/pakfire/pakfire > > index 9935481a5..0a144c517 100644 > > --- a/src/pakfire/pakfire > > +++ b/src/pakfire/pakfire > > @@ -258,51 +258,76 @@ > > &Pakfire::getcoredb("$force"); > > > > } elsif ("$ARGV[0]" eq "upgrade") { > > - my $use_color = ""; > > - my $reset_color = ""; > > - > > - if ("$Pakfire::enable_colors" eq "1") { > > - $reset_color = "$Pakfire::color{'normal'}"; > > - $use_color = > > "$Pakfire::color{'lightpurple'}"; > > - } > > - > > &Pakfire::message("CORE INFO: Checking for Core > > updates..."); > > - > > ### Make sure that the core db is not outdated. > > &Pakfire::getcoredb("noforce"); > > - my %coredb = &Pakfire::coredbinfo(); > > > > - if (defined $coredb{'AvailableRelease'}) { > > - &Pakfire::upgradecore(); > > - } else { > > - &Pakfire::message("CORE WARN: No new Core > > upgrades available. You are on release ".$coredb{'Release'}); > > - } > > + my %coredb = &Pakfire::coredbinfo(); > > + &Pakfire::message("CORE WARN: No new Core upgrades > > available. You are on release ".$coredb{'Release'}) unless (defined > > $coredb{'AvailableRelease'}); > > > > &Pakfire::message("PAKFIRE INFO: Checking for > > package updates..."); > > ### Make sure that the package list is not > > outdated. > > &Pakfire::dbgetlist("noforce"); > > > > my @deps = (); > > - if (my %upgradepaks = &Pakfire::dblist("upgrade")) > > { > > - # Resolve the dependencies of the to be > > upgraded packages > > - @deps = > > &Pakfire::resolvedeps_recursive(keys %upgradepaks); > > + my %upgradepaks = &Pakfire::dblist("upgrade"); > > > > - foreach $pak (sort keys %upgradepaks) { > > - print "${use_color}Update: > > $pak\nVersion: $upgradepaks{$pak}{'ProgVersion'} -> > > $upgradepaks{$pak}{'AvailableProgVersion'}\n"; > > - print "Release: > > $upgradepaks{$pak}{'Release'} -> > > $upgradepaks{$pak}{'AvailableRelease'}${reset_color}\n"; > > - } > > + # Resolve the dependencies of the to be upgraded > > packages > > + @deps = &Pakfire::resolvedeps_recursive(keys > > %upgradepaks) if (%upgradepaks); > > + &Pakfire::message("PAKFIRE WARN: No new package > > upgrades available.") unless (@deps); > > + > > + if (defined $coredb{'AvailableRelease'} || > > %upgradepaks) { > > + &Pakfire::message(""); > > &Pakfire::message(""); > > - &Pakfire::message("PAKFIRE UPGR: We are > > going to install all packages listed above."); > > + &Pakfire::message("PAKFIRE INFO: Upgrade > > summary:"); > > + &Pakfire::message(""); > > + > > + if (defined $coredb{'AvailableRelease'}) { > > + &Pakfire::message("CORE INFO: Core- > > Update $coredb{'CoreVersion'} to install:"); > > + &Pakfire::message("CORE UPGR: > > Release: $coredb{'Release'} -> $coredb{'AvailableRelease'}"); > > + &Pakfire::message(""); > > + } > > + > > + if (@deps) { > > + &Pakfire::message("PAKFIRE INFO: > > New dependencies to install:"); > > + my $totalsize = 0; > > + foreach $pak (@deps) { > > + unless (defined > > $upgradepaks{$pak} || &Pakfire::isinstalled($pak) == 0) { > > + my $size = > > &Pakfire::getsize("$pak"); > > + $totalsize += > > $size; > > + $size = > > &Pakfire::beautifysize($size); > > + &Pakfire::message(" > > PAKFIRE INFO: $pak \t - $size"); > > + } > > + } > > + $totalsize = > > &Pakfire::beautifysize($totalsize); > > + &Pakfire::message(""); > > + &Pakfire::message("PAKFIRE INFO: > > Total size: \t ~ $totalsize"); > > + &Pakfire::message(""); > > + } > > + > > + if (%upgradepaks) { > > + &Pakfire::message("PAKFIRE INFO: > > Packages to upgrade:"); > > + foreach $pak (sort keys > > %upgradepaks) { > > + &Pakfire::message("PAKFIRE > > UPGR: $pak\t$upgradepaks{$pak}{'ProgVersion'}- > > $upgradepaks{$pak}{'Release'} -> > > $upgradepaks{$pak}{'AvailableProgVersion'}- > > $upgradepaks{$pak}{'AvailableRelease'}"); > > + } > > + &Pakfire::message(""); > > + } > > + > > if ($interactive) { > > - &Pakfire::message("PAKFIRE INFO: Is this > > okay? [y/N]"); > > + &Pakfire::message("PAKFIRE INFO: Is > > this okay? [y/N]"); > > my $ret = <STDIN>; > > chomp($ret); > > &Pakfire::logger("PAKFIRE INFO: > > Answer: $ret");need > > if ( $ret ne "y" ) { > > - &Pakfire::message("PAKFIRE ERROR: > > Installation aborted."); > > - exit 1; > > + &Pakfire::message("PAKFIRE > > ERROR: Installation aborted."); > > + exit 1; > > } > > } > > + > > + # Perform core update > > + if (defined $coredb{'AvailableRelease'}) { > > + &Pakfire::upgradecore(); > > + } > > > > # Download packages > > foreach $pak (sort keys %upgradepaks) { > > @@ -323,8 +348,6 @@ > > foreach $pak (sort keys %upgradepaks) { > > &Pakfire::upgradepak("$pak"); > > } > > - } else { > > - &Pakfire::message("PAKFIRE WARN: No new > > package upgrades available."); > > } > > > > } elsif ("$ARGV[0]" eq "list") { > > -- > > 2.34.1 > > > > > > -- > > Dit bericht is gescanned op virussen en andere gevaarlijke > > inhoud door MailScanner en lijkt schoon te zijn. > > > >
Hello, > On 22 Mar 2022, at 18:32, Robin Roevens <robin.roevens@disroot.org> wrote: > > > Michael Tremer schreef op ma 21-03-2022 om 16:36 [+0000]: >> Hello, >> >>> On 9 Mar 2022, at 22:56, Robin Roevens <robin.roevens@disroot.org> >>> wrote: >>> >>> - Revamped 'pakfire update' more in the style of 'pakfire >>> install' for a more consistent look and feel. >>> - Add core update details to output >>> - Add new dependencies to install to output >>> - First collect all upgrade info, then perform core update >>> together with available package updates and after user >>> confirmation (without -y). >> >> I am sorry to disappoint Tom, but this is quite likely a no-can-do. >> >> The problem is that we are repeatedly tight on disk space which makes >> downloading everything first a bit of a problem. > > I'm not sure what you mean? Originally the upgrade function forced a > core upgrade first, then retrieved info about pak updates and requested > confirmation to install them. > I changed this into first retrieving all pak and core update info (not > the paks or core themselves), presenting the info to the user and > asking confirmation. After that the core is downloaded and installed, > then the paks are downloaded and installed. > So in terms of disk space usage, nothing should have changed except > that pak metadata is now downloaded before the core update is > performed. But those few text files should not be the problem I assume? Oh okay. I read this differently then. The text files do not matter when it comes to disk space, but: We sometimes change the tree in a core update (2.27 -> 2.29 or something), which this will break. Pakfire will then not find the correct new packages. The meta data needs to be updated after the core update is being installed and then pakfire can decide what packages to install. Otherwise this would become a two-step update now. > >> >> We do not have any mechanism that checks whether enough space for >> either the downloads or for extracting the update is available, which >> is why the old way was more of a step-by-step system. >> >> I don’t like the old system at all, but I would propose to add a >> function that checks if enough disk space is available before it asks >> the user whether they want to continue. >> >> That should consider: >> >> * The download size of the updates >> * The download size of all add-ons >> * The extracted size (for simplicity probably assuming that the core >> update is only adding new files) >> * The extracted size of all add-ons > > Great minds think alike, I suppose :-) I was already in the middle of > trying to implement size checking. But I didn't want the current > patchset submission to be postponed for that. (The last patch was > already some kind of 'extra' and not really that related to the purpose > of that patchset (the cleanup of dblist and using extra metadata)) Yes, we should approach things one by one. I have a massive work load at the moment and so reviewing large changes becomes a little bit difficult for me. > I already did want to check some things about the size calculation here > on the mailing list, but I figured that this patchset should first need > to be accepted, as the size calculation would depend on it. We can think about it and decide to implement it later... > But now that we are on this topic.. > I wanted to verify here if the size recorded in the meta-files, is the > size of the tar-files? This is the download size. > And since everything is tarred without compression (or so I think?) > this should more or less be equal to the installed size ? Or am I > mistaken? No, we compress the data. There are two tarballs inside each other. The outer tarball has the scripts and the payload and is not compressed. The payload is compressed using XZ. > If we ever decide to actually compress those we will need to record > uncompressed size in the metadata. So, the download size is usually quite far away from the installed size. ms@rice-oxley ~ % curl https://mirror1.ipfire.org/pakfire2/2.27/paks/samba-4.14.6-82.ipfire | gpg --decrypt | tar xO files.tar.xz | xz -d | wc -c % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 11.4M 100 11.4M 0 0 11.5M 0 --:--:-- --:--:-- --:--:-- 11.8M gpg: Signature made Tue 21 Dec 11:21:18 2021 GMT gpg: using RSA key 3ECA8AA4478208B924BB96206FEF7A8ED713594B gpg: Can't check signature: No public key 67164160 Samba is about 64MiB extracted (as tar) and 11 MiB downloaded. We would have to add this to the metadata so that you can use the installed size in your calculations. > > Then for calculation, I was thinking about separating temporary space > and installed space: > * core update: > -- Required tempspace: size of new core (tar-file) > -- required installspace: size of new core Agreed. > * new dependencies: > -- required tempspace: sum of size of all required deps (tar files) > -- required installspace: sum of size of all required deps Agreed. > * pak updates: > -- required tempspace: sum of size of all required paks (tar files) > -- required installspace: for each pak: size of installed pak version - > size of new pak version Technically you are correct. I am just unsure how much margin of error we should add to this. > in the end total required space is tempspace + installspace, if > installspace is positive. It can in theory also be negative (no core > update and size of updated paks is less than previous versions), then > required space = tempspace.. True. Again, rather be safe than sorry. > I'm not sure if downloaded pakfiles/core-updates are deleted after > install..? Or after some time ? I did not immediately find it in the > code. We could implement some cache pruning ('pakfire clear' or > 'pakfire clear-cache' or so?) Update.sh removes the downloaded core update before it starts extracting everything. So technically, it is on disk multiple times: 1) the downloaded file in /var/cache/pakfire 2) The extracted download file in /opt/pakfire/tmp 3) Then we extract the payload Worst case is that we need the downloaded space twice + the installed size. Pruning the cache is a bit difficult. Pakfire does not store the uninstall script which means that we need the package file when we uninstall a package. That is why we usually leave everything in the cache. An extra command would be helpful and pakfire should ideally delete the downloaded core update as soon as it got extracted (before calling update.sh). OMG this is complicated :) > this size diff at least for pak updates should be quite accurate, as > previous pak is uninstalled and new pak is installed during update. Any opened files will still remain on disk. Sometimes we replace things while they are running which won’t free disk space immediately. That is why I am suggestion some margin of error here of maybe 20%? I wouldn’t calculate this to the last byte. > I have been trying to calculate the actual size of a core update.. but > only just now I figured that a core update file is probably not a full > ipfire fs, but only the diff in files against previous core update :-) Yes it is. > For the actual size of a core update, we would need to have a diff of > the size of a clean ipfire on previous core and a clean ipfire on next > core.. I would just assume that all files are new files and will need extra space. We can’t easily compute the diff here. > But I don't think that is data we have already available or could > easily have available ? So assuming core update only adds files is > probably indeed the safest straight-forward method. Other method could > be to assume a percentage of new files-size in a core update..? We could, but what would that be? Sometimes we might extract 100M and only replace existing stuff. The we might add a new kernel which will 100M of modules on its own. I don’t think there is an easy formula. However, many systems have a / partition of only 2G. We need to be able to install updates to those systems cannot overshoot too much and hope for a gigabyte of space that we can use. My gut tells me that maybe taking the install size of the core update by 3/4 might be a good number? -Michael > > Robin > >> >> Then, we can even drop some last checks in the update scripts that >> aborted the update process if disk space was running low. >> >> How does that sound? >> >> -Michael >> >>> >>> Signed-off-by: Robin Roevens <robin.roevens@disroot.org> >>> --- >>> src/pakfire/pakfire | 79 +++++++++++++++++++++++++++++------------- >>> --- >>> 1 file changed, 51 insertions(+), 28 deletions(-) >>> >>> diff --git a/src/pakfire/pakfire b/src/pakfire/pakfire >>> index 9935481a5..0a144c517 100644 >>> --- a/src/pakfire/pakfire >>> +++ b/src/pakfire/pakfire >>> @@ -258,51 +258,76 @@ >>> &Pakfire::getcoredb("$force"); >>> >>> } elsif ("$ARGV[0]" eq "upgrade") { >>> - my $use_color = ""; >>> - my $reset_color = ""; >>> - >>> - if ("$Pakfire::enable_colors" eq "1") { >>> - $reset_color = "$Pakfire::color{'normal'}"; >>> - $use_color = >>> "$Pakfire::color{'lightpurple'}"; >>> - } >>> - >>> &Pakfire::message("CORE INFO: Checking for Core >>> updates..."); >>> - >>> ### Make sure that the core db is not outdated. >>> &Pakfire::getcoredb("noforce"); >>> - my %coredb = &Pakfire::coredbinfo(); >>> >>> - if (defined $coredb{'AvailableRelease'}) { >>> - &Pakfire::upgradecore(); >>> - } else { >>> - &Pakfire::message("CORE WARN: No new Core >>> upgrades available. You are on release ".$coredb{'Release'}); >>> - } >>> + my %coredb = &Pakfire::coredbinfo(); >>> + &Pakfire::message("CORE WARN: No new Core upgrades >>> available. You are on release ".$coredb{'Release'}) unless (defined >>> $coredb{'AvailableRelease'}); >>> >>> &Pakfire::message("PAKFIRE INFO: Checking for >>> package updates..."); >>> ### Make sure that the package list is not >>> outdated. >>> &Pakfire::dbgetlist("noforce"); >>> >>> my @deps = (); >>> - if (my %upgradepaks = &Pakfire::dblist("upgrade")) >>> { >>> - # Resolve the dependencies of the to be >>> upgraded packages >>> - @deps = >>> &Pakfire::resolvedeps_recursive(keys %upgradepaks); >>> + my %upgradepaks = &Pakfire::dblist("upgrade"); >>> >>> - foreach $pak (sort keys %upgradepaks) { >>> - print "${use_color}Update: >>> $pak\nVersion: $upgradepaks{$pak}{'ProgVersion'} -> >>> $upgradepaks{$pak}{'AvailableProgVersion'}\n"; >>> - print "Release: >>> $upgradepaks{$pak}{'Release'} -> >>> $upgradepaks{$pak}{'AvailableRelease'}${reset_color}\n"; >>> - } >>> + # Resolve the dependencies of the to be upgraded >>> packages >>> + @deps = &Pakfire::resolvedeps_recursive(keys >>> %upgradepaks) if (%upgradepaks); >>> + &Pakfire::message("PAKFIRE WARN: No new package >>> upgrades available.") unless (@deps); >>> + >>> + if (defined $coredb{'AvailableRelease'} || >>> %upgradepaks) { >>> + &Pakfire::message(""); >>> &Pakfire::message(""); >>> - &Pakfire::message("PAKFIRE UPGR: We are >>> going to install all packages listed above."); >>> + &Pakfire::message("PAKFIRE INFO: Upgrade >>> summary:"); >>> + &Pakfire::message(""); >>> + >>> + if (defined $coredb{'AvailableRelease'}) { >>> + &Pakfire::message("CORE INFO: Core- >>> Update $coredb{'CoreVersion'} to install:"); >>> + &Pakfire::message("CORE UPGR: >>> Release: $coredb{'Release'} -> $coredb{'AvailableRelease'}"); >>> + &Pakfire::message(""); >>> + } >>> + >>> + if (@deps) { >>> + &Pakfire::message("PAKFIRE INFO: >>> New dependencies to install:"); >>> + my $totalsize = 0; >>> + foreach $pak (@deps) { >>> + unless (defined >>> $upgradepaks{$pak} || &Pakfire::isinstalled($pak) == 0) { >>> + my $size = >>> &Pakfire::getsize("$pak"); >>> + $totalsize += >>> $size; >>> + $size = >>> &Pakfire::beautifysize($size); >>> + &Pakfire::message(" >>> PAKFIRE INFO: $pak \t - $size"); >>> + } >>> + } >>> + $totalsize = >>> &Pakfire::beautifysize($totalsize); >>> + &Pakfire::message(""); >>> + &Pakfire::message("PAKFIRE INFO: >>> Total size: \t ~ $totalsize"); >>> + &Pakfire::message(""); >>> + } >>> + >>> + if (%upgradepaks) { >>> + &Pakfire::message("PAKFIRE INFO: >>> Packages to upgrade:"); >>> + foreach $pak (sort keys >>> %upgradepaks) { >>> + &Pakfire::message("PAKFIRE >>> UPGR: $pak\t$upgradepaks{$pak}{'ProgVersion'}- >>> $upgradepaks{$pak}{'Release'} -> >>> $upgradepaks{$pak}{'AvailableProgVersion'}- >>> $upgradepaks{$pak}{'AvailableRelease'}"); >>> + } >>> + &Pakfire::message(""); >>> + } >>> + >>> if ($interactive) { >>> - &Pakfire::message("PAKFIRE INFO: Is this >>> okay? [y/N]"); >>> + &Pakfire::message("PAKFIRE INFO: Is >>> this okay? [y/N]"); >>> my $ret = <STDIN>; >>> chomp($ret); >>> &Pakfire::logger("PAKFIRE INFO: >>> Answer: $ret");need >>> if ( $ret ne "y" ) { >>> - &Pakfire::message("PAKFIRE ERROR: >>> Installation aborted."); >>> - exit 1; >>> + &Pakfire::message("PAKFIRE >>> ERROR: Installation aborted."); >>> + exit 1; >>> } >>> } >>> + >>> + # Perform core update >>> + if (defined $coredb{'AvailableRelease'}) { >>> + &Pakfire::upgradecore(); >>> + } >>> >>> # Download packages >>> foreach $pak (sort keys %upgradepaks) { >>> @@ -323,8 +348,6 @@ >>> foreach $pak (sort keys %upgradepaks) { >>> &Pakfire::upgradepak("$pak"); >>> } >>> - } else { >>> - &Pakfire::message("PAKFIRE WARN: No new >>> package upgrades available."); >>> } >>> >>> } elsif ("$ARGV[0]" eq "list") { >>> -- >>> 2.34.1 >>> >>> >>> -- >>> Dit bericht is gescanned op virussen en andere gevaarlijke >>> inhoud door MailScanner en lijkt schoon te zijn. >>> >> >> > > -- > Dit bericht is gescanned op virussen en andere gevaarlijke > inhoud door MailScanner en lijkt schoon te zijn.
diff --git a/src/pakfire/pakfire b/src/pakfire/pakfire index 9935481a5..0a144c517 100644 --- a/src/pakfire/pakfire +++ b/src/pakfire/pakfire @@ -258,51 +258,76 @@ &Pakfire::getcoredb("$force"); } elsif ("$ARGV[0]" eq "upgrade") { - my $use_color = ""; - my $reset_color = ""; - - if ("$Pakfire::enable_colors" eq "1") { - $reset_color = "$Pakfire::color{'normal'}"; - $use_color = "$Pakfire::color{'lightpurple'}"; - } - &Pakfire::message("CORE INFO: Checking for Core updates..."); - ### Make sure that the core db is not outdated. &Pakfire::getcoredb("noforce"); - my %coredb = &Pakfire::coredbinfo(); - if (defined $coredb{'AvailableRelease'}) { - &Pakfire::upgradecore(); - } else { - &Pakfire::message("CORE WARN: No new Core upgrades available. You are on release ".$coredb{'Release'}); - } + my %coredb = &Pakfire::coredbinfo(); + &Pakfire::message("CORE WARN: No new Core upgrades available. You are on release ".$coredb{'Release'}) unless (defined $coredb{'AvailableRelease'}); &Pakfire::message("PAKFIRE INFO: Checking for package updates..."); ### Make sure that the package list is not outdated. &Pakfire::dbgetlist("noforce"); my @deps = (); - if (my %upgradepaks = &Pakfire::dblist("upgrade")) { - # Resolve the dependencies of the to be upgraded packages - @deps = &Pakfire::resolvedeps_recursive(keys %upgradepaks); + my %upgradepaks = &Pakfire::dblist("upgrade"); - foreach $pak (sort keys %upgradepaks) { - print "${use_color}Update: $pak\nVersion: $upgradepaks{$pak}{'ProgVersion'} -> $upgradepaks{$pak}{'AvailableProgVersion'}\n"; - print "Release: $upgradepaks{$pak}{'Release'} -> $upgradepaks{$pak}{'AvailableRelease'}${reset_color}\n"; - } + # Resolve the dependencies of the to be upgraded packages + @deps = &Pakfire::resolvedeps_recursive(keys %upgradepaks) if (%upgradepaks); + &Pakfire::message("PAKFIRE WARN: No new package upgrades available.") unless (@deps); + + if (defined $coredb{'AvailableRelease'} || %upgradepaks) { + &Pakfire::message(""); &Pakfire::message(""); - &Pakfire::message("PAKFIRE UPGR: We are going to install all packages listed above."); + &Pakfire::message("PAKFIRE INFO: Upgrade summary:"); + &Pakfire::message(""); + + if (defined $coredb{'AvailableRelease'}) { + &Pakfire::message("CORE INFO: Core-Update $coredb{'CoreVersion'} to install:"); + &Pakfire::message("CORE UPGR: Release: $coredb{'Release'} -> $coredb{'AvailableRelease'}"); + &Pakfire::message(""); + } + + if (@deps) { + &Pakfire::message("PAKFIRE INFO: New dependencies to install:"); + my $totalsize = 0; + foreach $pak (@deps) { + unless (defined $upgradepaks{$pak} || &Pakfire::isinstalled($pak) == 0) { + my $size = &Pakfire::getsize("$pak"); + $totalsize += $size; + $size = &Pakfire::beautifysize($size); + &Pakfire::message("PAKFIRE INFO: $pak \t - $size"); + } + } + $totalsize = &Pakfire::beautifysize($totalsize); + &Pakfire::message(""); + &Pakfire::message("PAKFIRE INFO: Total size: \t ~ $totalsize"); + &Pakfire::message(""); + } + + if (%upgradepaks) { + &Pakfire::message("PAKFIRE INFO: Packages to upgrade:"); + foreach $pak (sort keys %upgradepaks) { + &Pakfire::message("PAKFIRE UPGR: $pak\t$upgradepaks{$pak}{'ProgVersion'}-$upgradepaks{$pak}{'Release'} -> $upgradepaks{$pak}{'AvailableProgVersion'}-$upgradepaks{$pak}{'AvailableRelease'}"); + } + &Pakfire::message(""); + } + if ($interactive) { - &Pakfire::message("PAKFIRE INFO: Is this okay? [y/N]"); + &Pakfire::message("PAKFIRE INFO: Is this okay? [y/N]"); my $ret = <STDIN>; chomp($ret); &Pakfire::logger("PAKFIRE INFO: Answer: $ret"); if ( $ret ne "y" ) { - &Pakfire::message("PAKFIRE ERROR: Installation aborted."); - exit 1; + &Pakfire::message("PAKFIRE ERROR: Installation aborted."); + exit 1; } } + + # Perform core update + if (defined $coredb{'AvailableRelease'}) { + &Pakfire::upgradecore(); + } # Download packages foreach $pak (sort keys %upgradepaks) { @@ -323,8 +348,6 @@ foreach $pak (sort keys %upgradepaks) { &Pakfire::upgradepak("$pak"); } - } else { - &Pakfire::message("PAKFIRE WARN: No new package upgrades available."); } } elsif ("$ARGV[0]" eq "list") {