From patchwork Sat Mar 30 11:35:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Schantl X-Patchwork-Id: 7699 Return-Path: Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) client-signature ECDSA (secp384r1)) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by web04.haj.ipfire.org (Postfix) with ESMTPS id 4V6FZx6h29z3wwl for ; Sat, 30 Mar 2024 11:35:53 +0000 (UTC) Received: from mail02.haj.ipfire.org (mail02.haj.ipfire.org [172.28.1.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) client-signature ECDSA (secp384r1)) (Client CN "mail02.haj.ipfire.org", Issuer "R3" (verified OK)) by mail01.ipfire.org (Postfix) with ESMTPS id 4V6FZw4L0Rz2jq; Sat, 30 Mar 2024 11:35:52 +0000 (UTC) Received: from mail02.haj.ipfire.org (localhost [127.0.0.1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4V6FZv2vQRz32gp; Sat, 30 Mar 2024 11:35:51 +0000 (UTC) Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) client-signature ECDSA (secp384r1)) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4V6FZq0vXqz30CD for ; Sat, 30 Mar 2024 11:35:47 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by mail01.ipfire.org (Postfix) with ESMTPSA id 4V6FZp4FXWz1Zn; Sat, 30 Mar 2024 11:35:46 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003ed25519; t=1711798546; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=drJTB1RR+hfUhwMBNvG0OGWHOuPPck4NqDqBd9qSfm4=; b=/X8wvsV4iAyGIeZAreB5PNWsR8hkksqY4m6rEkA465X+s18ZDSpUcypg9lmtgtpvS8CS2y Gb+pevLSVzO1RZCw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003rsa; t=1711798546; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=drJTB1RR+hfUhwMBNvG0OGWHOuPPck4NqDqBd9qSfm4=; b=Fp+FO2obHUcdfLfe9NXm7rxiPQdn12SFYi45R/TKxdDK4y3szL6aNy/MiXUs1IITNG5R1c +5975H5mJMZk//zhQULgAFRv6e87r5RD0KaXnPe5jrI22mXXAMyQUvKR6BHKoi73bJk/5i +FBSkGWrtQ+RUBv2Qg1logGWLG3rLB+nyAzQ6ZHeGvwRDo660ZzAz7AXutVYpyYU4KEBIp 6c1oEfzA8RepU26cJMDArtovHdyNkDfi/MHvSCcc2Sa78tRb/OpTro+f2SjGdOSSIk0y7b EpMFCDvtS1RJ1RAiebXy2xPqosMvPJmFnuLwjABtEEgsWYAlKgvd535g/+mNFw== From: Stefan Schantl To: development@lists.ipfire.org Subject: [PATCH 5/5] ids-functions.pl: Use libarchive to extract archives Date: Sat, 30 Mar 2024 12:35:30 +0100 Message-Id: <20240330113530.1710702-5-stefan.schantl@ipfire.org> In-Reply-To: <20240330113530.1710702-1-stefan.schantl@ipfire.org> References: <20240330113530.1710702-1-stefan.schantl@ipfire.org> MIME-Version: 1.0 Message-ID-Hash: 5IWNXABDDUIENGFNOYICC5SJZZLPKOQS X-Message-ID-Hash: 5IWNXABDDUIENGFNOYICC5SJZZLPKOQS X-MailFrom: stefan.schantl@ipfire.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.8 Precedence: list List-Id: IPFire development talk Archived-At: <> List-Archive: <> List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: This gives us a lot of benefits: * Speed up the extraction process * More supported archive types due the power of libarchive * Support of passphrase protected archives It also fixes a problem with non extracted files next to a zero sized file inside an archive. Fixes #13632. Signed-off-by: Stefan Schantl --- config/cfgroot/ids-functions.pl | 56 +++++++++++---------------------- 1 file changed, 18 insertions(+), 38 deletions(-) diff --git a/config/cfgroot/ids-functions.pl b/config/cfgroot/ids-functions.pl index c29a5151f..3eb883aa9 100644 --- a/config/cfgroot/ids-functions.pl +++ b/config/cfgroot/ids-functions.pl @@ -30,7 +30,7 @@ require "${General::swroot}/network-functions.pl"; require "${General::swroot}/suricata/ruleset-sources"; # Load perl module to deal with Archives. -use Archive::Tar; +use Archive::Peek::Libarchive; # Load perl module to deal with files and path. use File::Basename; @@ -515,9 +515,6 @@ sub downloadruleset ($) { sub extractruleset ($) { my ($provider) = @_; - # Disable chown functionality when uncompressing files. - $Archive::Tar::CHOWN = "0"; - # Get full path and downloaded rulesfile for the given provider. my $tarball = &_get_dl_rulesfile($provider); @@ -547,13 +544,11 @@ sub extractruleset ($) { } elsif ( $type eq "archive") { # Initialize the tar module. - my $tar = Archive::Tar->new($tarball); - - # Get the filelist inside the tarball. - my @packed_files = $tar->list_files; + my $tar = Archive::Peek::Libarchive->new(filename => $tarball); - # Loop through the filelist. - foreach my $packed_file (@packed_files) { + # Loop through the archive + $tar->iterate( sub { + my ($packed_file, $content) = @_; my $destination; # Splitt the packed file into chunks. @@ -572,13 +567,13 @@ sub extractruleset ($) { # Handle rules files. } elsif ($file =~ m/\.rules$/) { # Skip rule files which are not located in the rules directory or archive root. - next unless(($packed_file =~ /^rules\//) || ($packed_file =~ /^$provider-rules\//) || ($packed_file !~ /\//)); + return unless(($packed_file =~ /^rules\//) || ($packed_file =~ /^$provider-rules\//) || ($packed_file !~ /\//)); # Skip deleted.rules. # # Mostly they have been taken out for correctness or performance reasons and therfore # it is not a great idea to enable any of them. - next if($file =~ m/deleted.rules$/); + return if($file =~ m/deleted.rules$/); my $rulesfilename; @@ -615,39 +610,24 @@ sub extractruleset ($) { $destination = "$tmp_rules_directory/$rulesfilename"; } else { # Skip all other files. - next; + return; } # Check if the destination file exists. unless(-e "$destination") { - # Extract the file to the temporary directory. - $tar->extract_file("$packed_file", "$destination"); + # Open filehandle to write the content to a new file. + open(FILE, ">", "$destination") or die "Could not open $destination. $!\n"; } else { - # Generate temporary file name, located in the temporary rules directory and a suffix of ".tmp". - my $tmp = File::Temp->new( SUFFIX => ".tmp", DIR => "$tmp_rules_directory", UNLINK => 0 ); - my $tmpfile = $tmp->filename(); - - # Extract the file to the new temporary file name. - $tar->extract_file("$packed_file", "$tmpfile"); - - # Open the the existing file. - open(DESTFILE, ">>", "$destination") or die "Could not open $destination. $!\n"; - open(TMPFILE, "<", "$tmpfile") or die "Could not open $tmpfile. $!\n"; - - # Loop through the content of the temporary file. - while () { - # Append the content line by line to the destination file. - print DESTFILE "$_"; - } + # Open filehandle to append the content to the existing file. + open(FILE, ">>", "$destination") or die "Could not open $destination. $!\n"; + } - # Close the file handles. - close(TMPFILE); - close(DESTFILE); + # Write the extracted file content to the filehandle. + print FILE "$content" if ($content); - # Remove the temporary file. - unlink("$tmpfile"); - } - } + # Close the file handle. + close(FILE); + }); } }