From patchwork Wed Feb 23 20:21:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robin Roevens X-Patchwork-Id: 5276 Return-Path: Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by web04.haj.ipfire.org (Postfix) with ESMTPS id 4K3nWM010tz3xgY for ; Wed, 23 Feb 2022 20:21:51 +0000 (UTC) Received: from mail02.haj.ipfire.org (mail02.haj.ipfire.org [172.28.1.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail02.haj.ipfire.org", Issuer "R3" (verified OK)) by mail01.ipfire.org (Postfix) with ESMTPS id 4K3nWK6hTrz2lS; Wed, 23 Feb 2022 20:21:49 +0000 (UTC) Received: from mail02.haj.ipfire.org (localhost [127.0.0.1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4K3nWK6K8qz2ypl; Wed, 23 Feb 2022 20:21:49 +0000 (UTC) Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4K3nWJ70Wfz2xS1 for ; Wed, 23 Feb 2022 20:21:48 +0000 (UTC) Received: from knopi.disroot.org (knopi.disroot.org [178.21.23.139]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mail01.ipfire.org (Postfix) with ESMTPS id 4K3nWJ4CBMz5Vn for ; Wed, 23 Feb 2022 20:21:48 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by disroot.org (Postfix) with ESMTP id 43FC88DBB8 for ; Wed, 23 Feb 2022 21:21:48 +0100 (CET) X-Virus-Scanned: SPAM Filter at disroot.org Received: from knopi.disroot.org ([127.0.0.1]) by localhost (disroot.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id cE4aaWr_zoDP for ; Wed, 23 Feb 2022 21:21:46 +0100 (CET) Received: from chojin.sicho.home (amaterasu.sicho.home [192.168.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (no client certificate requested) (Authenticated sender) by hachiman (MailScanner Milter) with SMTP id 0E200152BB; Wed, 23 Feb 2022 21:21:39 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=disroot.org; s=mail; t=1645647706; bh=ZYXj0DJHjjIJCxPtJSl4irN/QDTgYYodPjho3NHJyro=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=EGBOwTA2fQPtFwdLX0XzfoQgyqjOfpfcLTWOfKsCY3JN0E1fY4OGpfI6fMY+uQ3RN wx2eVMZDz5bzyXFgTFsNQDfdoIoyxSl/NDVng00gS28eup40NbI9GvOtKSqDcNquTP iL5UqcqoAA3DsQYy+KrnkewCTHl9pTuw9LPGa9I/c3HdTnONHrTnkl4FeeO8DMKINH fVFOI9uwe/TmT4PYiyvdiit9qVkyUr6OuFoox5AcxrJkoxjWx3/iXvAdXe+61FIngn Dc0ygQyKvv8RJc9lxXOj85/uKR/Dj7eXoBVcp3hESlOmSCpy3QIq+rI01MmBxnME8m gGYPuxRNa/pTw== From: Robin Roevens To: development@lists.ipfire.org Subject: [PATCH] pakfire: Better errorhandling on downloads Date: Wed, 23 Feb 2022 21:21:30 +0100 Message-Id: <20220223202130.6104-2-robin.roevens@disroot.org> In-Reply-To: <20220223202130.6104-1-robin.roevens@disroot.org> References: <20220223202130.6104-1-robin.roevens@disroot.org> Mime-Version: 1.0 X-sicho-MailScanner-ID: 0E200152BB.A8A80 X-sicho-MailScanner: Found to be clean X-sicho-MailScanner-From: robin.roevens@disroot.org X-sicho-MailScanner-Watermark: 1646252500.15245@d4ZrvSK1AVZQ88hkF3XFtg ARC-Seal: i=1; s=202003rsa; d=lists.ipfire.org; t=1645647708; a=rsa-sha256; cv=none; b=G7hsM/FpFoPdst9CbZMBsPwnuH2U0rzu73g8TJBoLw47khCHomJMnSF8zXow8mMBvs7YwZ V8ABAbY4bWn2DxAkwbKtlLeNXTHFKOXItV64Uxn8lo216cI1PREpb6uylo7suH2a8ntCTF 33I260zoLb8DKFzC/ZpO+GJIQ3P9vmwN6sjYjw3MO5hVqt2I99jTawc4PIfLXnmAVT1WG4 5I5L7Wjgpt0gSa14uref1E1Aos6OOT24XO/Q28750gw/Ml8WdyRXx3f8UGO6cJajwjdCMh 6nY3SyOniEun2xSNf3fIcDzEiS5RDVE1auuIXIx0O2Lu9igFHnuFTkhW+4Uorg== ARC-Authentication-Results: i=1; mail01.ipfire.org; dkim=pass header.d=disroot.org header.s=mail header.b=EGBOwTA2; dmarc=pass (policy=quarantine) header.from=disroot.org; spf=pass (mail01.ipfire.org: domain of robin.roevens@disroot.org designates 178.21.23.139 as permitted sender) smtp.mailfrom=robin.roevens@disroot.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=lists.ipfire.org; s=202003rsa; t=1645647708; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LJ0kjUKyqLjBoKNTkXCmzvx/mJBgiFWL02MTXRDBCHQ=; b=GNpPJR1Zw9zAVJ1PhpjkenUbZt/zZafBLu8u1zfc/QIcfmOUQB05a8eVZWoWbkViO4nnbT x7GC3Bk9FQyWMVzQuiwxD1hUdH+BbhSb6db7K14mW0gEoywFwbckVRmt6/msyA1sLJblJb 0PpZgeaxVhH9+mbusV09cMgQ8Mc2G4Ou09VsYYKFfu9k2efxkHOZ03MhcL1tXf3Z+PA88C FCvQIuUe2TrYa0qL/Q3StGjXTaI3nvzVOQGXrj70pSwhp7vWx//dLs1SkPMPa8uV2TUdPF C1mJhAZrlfNm5/W1Qq1qX0IAD+q+HfrqeIL1g2EvPjQTEfVea3YisH+g5KJ+Hw== Authentication-Results: mail01.ipfire.org; dkim=pass header.d=disroot.org header.s=mail header.b=EGBOwTA2; dmarc=pass (policy=quarantine) header.from=disroot.org; spf=pass (mail01.ipfire.org: domain of robin.roevens@disroot.org designates 178.21.23.139 as permitted sender) smtp.mailfrom=robin.roevens@disroot.org X-Rspamd-Server: mail01.haj.ipfire.org X-Spamd-Result: default: False [-8.78 / 11.00]; REPLY(-4.00)[]; BAYES_HAM(-3.00)[99.99%]; IP_REPUTATION_HAM(-1.16)[asn: 50673(-0.33), country: NL(-0.01), ip: 178.21.23.139(-0.82)]; MID_CONTAINS_FROM(1.00)[]; NEURAL_HAM(-0.99)[-0.990]; SPF_REPUTATION_HAM(-0.63)[-0.62650601313057]; R_MISSING_CHARSET(0.50)[]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[disroot.org,quarantine]; R_SPF_ALLOW(-0.20)[+a:c]; R_DKIM_ALLOW(-0.20)[disroot.org:s=mail]; MIME_GOOD(-0.10)[text/plain]; MX_GOOD(-0.01)[]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[development@lists.ipfire.org]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; ARC_SIGNED(0.00)[lists.ipfire.org:s=202003rsa:i=1]; RCVD_COUNT_THREE(0.00)[4]; ASN(0.00)[asn:50673, ipnet:178.21.23.0/24, country:NL]; DKIM_TRACE(0.00)[disroot.org:+]; MIME_TRACE(0.00)[0:+]; RCPT_COUNT_TWO(0.00)[2]; TO_DN_SOME(0.00)[] X-Rspamd-Queue-Id: 4K3nWJ4CBMz5Vn X-BeenThere: development@lists.ipfire.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: IPFire development talk List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: development-bounces@lists.ipfire.org Sender: "Development" - Add true/false return codes to fetchfile, getmetafile and getmirrors indicating succes or failure. - Check on those return codes and fail gracefully with clean error message(s) when downloads fail. - Replace duplicate meta-file fetching code in dbgetlist with getmetafile function (fixing possibly missed cariage return conversion in meta-files). - Remove pointless 5 retries to download server-list.db in selectmirror as fetchfile already retries 5 times. --- src/pakfire/lib/functions.pl | 84 +++++++++++++++++++++--------------- 1 file changed, 49 insertions(+), 35 deletions(-) diff --git a/src/pakfire/lib/functions.pl b/src/pakfire/lib/functions.pl index d4e338f23..24c55fd4a 100644 --- a/src/pakfire/lib/functions.pl +++ b/src/pakfire/lib/functions.pl @@ -2,7 +2,7 @@ ############################################################################### # # # IPFire.org - A linux based firewall # -# Copyright (C) 2007-2021 IPFire Team # +# Copyright (C) 2007-2022 IPFire Team # # # # This program is free software: you can redistribute it and/or modify # # it under the terms of the GNU General Public License as published by # @@ -206,7 +206,7 @@ sub fetchfile { if ( $code eq "500" ) { message("Giving up: There was no chance to get the file \"$getfile\" from any available server.\nThere was an error on the way. Please fix it."); - return 1; + return 0; } if ($response->is_success) { @@ -226,7 +226,7 @@ sub fetchfile { } logger("DOWNLOAD FINISHED: $file"); $allok = 1; - return 0; + return 1; } else { logger("DOWNLOAD ERROR: Could not open $Conf::tmpdir/$bfile for writing."); } @@ -235,7 +235,7 @@ sub fetchfile { } } message("DOWNLOAD ERROR: There was no chance to get the file \"$getfile\" from any available server.\nMay be you should run \"pakfire update\" to get some new servers."); - return 1; + return 0; } sub getmirrors { @@ -256,9 +256,14 @@ sub getmirrors { } if ("$force" eq "force") { - fetchfile("$Conf::version/lists/server-list.db", "$Conf::mainserver"); - move("$Conf::cachedir/server-list.db", "$Conf::dbdir/lists/server-list.db"); + if (fetchfile("$Conf::version/lists/server-list.db", "$Conf::mainserver")) { + move("$Conf::cachedir/server-list.db", "$Conf::dbdir/lists/server-list.db"); + } elsif (! -e "$Conf::dbdir/lists/server-list.db" ) { + # if we end up with no server-list at all, return failure + return 0; + } } + return 1; } sub getcoredb { @@ -279,8 +284,9 @@ sub getcoredb { } if ("$force" eq "force") { - fetchfile("lists/core-list.db", ""); - move("$Conf::cachedir/core-list.db", "$Conf::dbdir/lists/core-list.db"); + if (fetchfile("lists/core-list.db", "")) { + move("$Conf::cachedir/core-list.db", "$Conf::dbdir/lists/core-list.db"); + } } } @@ -318,15 +324,13 @@ sub selectmirror { ### Check if there is a current server list and read it. # If there is no list try to get one. - my $count = 0; - while (!(open(FILE, "<$Conf::dbdir/lists/server-list.db")) && ($count lt 5)) { - $count++; - getmirrors("noforce"); - } - if ($count == 5) { - message("MIRROR ERROR: Could not find or download a server list"); - exit 1; + unless (open(FILE, "<$Conf::dbdir/lists/server-list.db")) { + unless (getmirrors("noforce")) { + message("MIRROR ERROR: Could not find or download a server list"); + exit 1; + } } + my @lines = ; close(FILE); @@ -390,8 +394,13 @@ sub dbgetlist { } if ("$force" eq "force") { - fetchfile("lists/packages_list.db", ""); - move("$Conf::cachedir/packages_list.db", "$Conf::dbdir/lists/packages_list.db"); + if (fetchfile("lists/packages_list.db", "")) { + move("$Conf::cachedir/packages_list.db", "$Conf::dbdir/lists/packages_list.db"); + } elsif ( -e "$Conf::dbdir/lists/packages_list.db" ) { + # If we end up with no db file after download error there + # is nothing more we can do here. + return 0; + } } # Update the meta database if new packages was in the package list @@ -419,8 +428,7 @@ sub dbgetlist { @templine = split(/\;/,$prog); if (("$metadata{'Name'}" eq "$templine[0]") && ("$metadata{'Release'}" ne "$templine[2]")) { move("$Conf::dbdir/meta/meta-$metadata{'Name'}","$Conf::dbdir/meta/old_meta-$metadata{'Name'}"); - fetchfile("meta/meta-$metadata{'Name'}", ""); - move("$Conf::cachedir/meta-$metadata{'Name'}", "$Conf::dbdir/meta/meta-$metadata{'Name'}"); + getmetafile($metadata{'Name'}); } } } @@ -532,11 +540,14 @@ sub dblist { sub resolvedeps_one { my $pak = shift; - - getmetafile("$pak"); - + message("PAKFIRE RESV: $pak: Resolving dependencies..."); + unless (getmetafile("$pak")) { + message("PAKFIRE ERROR: Error retrieving dependency information on $pak. Unable to resolve dependencies."); + exit 1; + }; + my %metadata = parsemetafile("$Conf::dbdir/meta/meta-$pak"); my @all; my @deps = split(/ /, $metadata{'Dependencies'}); @@ -629,14 +640,10 @@ sub cleanup { sub getmetafile { my $pak = shift; - - unless ( -e "$Conf::dbdir/meta/meta-$pak" ) { - fetchfile("meta/meta-$pak", ""); - move("$Conf::cachedir/meta-$pak", "$Conf::dbdir/meta/meta-$pak"); - } - - if ( -z "$Conf::dbdir/meta/meta-$pak" ) { - fetchfile("meta/meta-$pak", ""); + + # Try to download meta-file if we don't have one yet, or it is empty for some reason + if ((! -e "$Conf::dbdir/meta/meta-$pak" ) || ( -z "$Conf::dbdir/meta/meta-$pak" )) { + return 0 unless (fetchfile("meta/meta-$pak", "")); move("$Conf::cachedir/meta-$pak", "$Conf::dbdir/meta/meta-$pak"); } @@ -651,6 +658,7 @@ sub getmetafile { print FILE $string; } close(FILE); + return 1; } @@ -713,8 +721,11 @@ sub getpak { my $pak = shift; my $force = shift; - getmetafile("$pak"); - + unless (getmetafile("$pak")) { + message("PAKFIRE ERROR: Unable to retrieve $pak metadata."); + exit 1; + } + my %metadata = parsemetafile("$Conf::dbdir/meta/meta-$pak"); my $file = $metadata{'File'}; @@ -728,8 +739,11 @@ sub getpak { return $file; } } - - fetchfile("paks/$file", ""); + + unless (fetchfile("paks/$file", "")) { + message("PAKFIRE ERROR: Unable to download $pak."); + exit 1; + } return $file; }