From patchwork Tue Mar 22 19:40:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Schantl X-Patchwork-Id: 5386 Return-Path: Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by web04.haj.ipfire.org (Postfix) with ESMTPS id 4KNML15HwBz3xqd for ; Tue, 22 Mar 2022 19:41:13 +0000 (UTC) Received: from mail02.haj.ipfire.org (mail02.haj.ipfire.org [172.28.1.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail02.haj.ipfire.org", Issuer "R3" (verified OK)) by mail01.ipfire.org (Postfix) with ESMTPS id 4KNMKy4DMtz5V3; Tue, 22 Mar 2022 19:41:10 +0000 (UTC) Received: from mail02.haj.ipfire.org (localhost [127.0.0.1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4KNMKx4Bf4z2yxs; Tue, 22 Mar 2022 19:41:09 +0000 (UTC) Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4KNMKv5K91z2xWt for ; Tue, 22 Mar 2022 19:41:07 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by mail01.ipfire.org (Postfix) with ESMTPSA id 4KNMKv2rpyzLX; Tue, 22 Mar 2022 19:41:07 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003ed25519; t=1647978067; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=uMxCrqyDrzrfCF/FVN1UL4sgIdLUK1vkwKaR7OmvQ4k=; b=CWgC1ZqFL87lF3NspdSt1OmyNbYLhKFDCKTh2WUF6lKtDKoyDEUzfx58TxJKLM9T9crUyX 4Vq3M2mHp+hyTzBw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003rsa; t=1647978067; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=uMxCrqyDrzrfCF/FVN1UL4sgIdLUK1vkwKaR7OmvQ4k=; b=fzdx27JIPhndXb+LmWYM+vcu1HC/BbRpb5O7x5Y3TEPiyPTb+GR2AIyhxyWQlC2sL1J4g6 BpPoEeouZgjR0GsftJGLiiojFDsIofaqwh+xe5NKCj3bOYeRW+tu+BM7lBaQzJCwHHgovb sdZKKdP/42Hwa0RoEBNLphjbW62EGtoNdJy4RRfDy0wtWA6p+O8BdMGZqQAClmdfFig3gT ui/9Ve4r5oOzju5ZCW7Km4VHOFih/8Gsd3nlBdMAl8C9uDkgQlQgIvjO6hS8D9bZbSKbrP +LwEhuAmUuUTlERg7y7HKx/us1l4JLc0SKVhzZc463qnBVxs/CNPQ5bNsp17VQ== From: Stefan Schantl To: development@lists.ipfire.org Subject: [PATCH 4/5] ids-functions.pl: Use If-Modified-Since header to reduce file downloads. Date: Tue, 22 Mar 2022 20:40:58 +0100 Message-Id: <20220322194059.3030-4-stefan.schantl@ipfire.org> In-Reply-To: <20220322194059.3030-1-stefan.schantl@ipfire.org> References: <20220322194059.3030-1-stefan.schantl@ipfire.org> MIME-Version: 1.0 X-BeenThere: development@lists.ipfire.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: IPFire development talk List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: development-bounces@lists.ipfire.org Sender: "Development" When using the "If-Modified-Since" header, the server can be requested if a modified version of the file can be served. In case that is true, the file will be sent and stored by the downloader function. If the file has not been touched since the last time, the server will respond with the code "304" (Not modified). This tells us, that the current stored file is the latest one (still up-to-date) and we safely can skip the download attempt for this provider. Signed-off-by: Stefan Schantl --- config/cfgroot/ids-functions.pl | 38 ++++++++++++++++++++++++++++++--- 1 file changed, 35 insertions(+), 3 deletions(-) diff --git a/config/cfgroot/ids-functions.pl b/config/cfgroot/ids-functions.pl index dfbeb1a7d..d7df41dd1 100644 --- a/config/cfgroot/ids-functions.pl +++ b/config/cfgroot/ids-functions.pl @@ -365,9 +365,25 @@ sub downloadruleset ($) { my $tmp = File::Temp->new( SUFFIX => ".tmp", DIR => "/var/tmp/", UNLINK => 0 ); my $tmpfile = $tmp->filename(); + # Genarate and assign file name and path to store the downloaded rules file. + my $dl_rulesfile = &_get_dl_rulesfile($provider); + + # Load perl module to deal with file atributes. + use File::stat; + + # Get the mtime of the rulesfile if it exists. + my $mtime = (stat($dl_rulesfile)->mtime) if (-f $dl_rulesfile); + + # Convert the mtime into gmtime format. + my $gmtime = gmtime($mtime || 0); + # Pass the requested url to the downloader. my $request = HTTP::Request->new(GET => $url); + # Add the If-Modified-Since header to the request, containing the omited and converted + # mtime of the downloaded rules file, if one is present. + $request->header( 'If-Modified-Since' => "$gmtime" ); + my $dl_attempt = 1; my $response; @@ -381,6 +397,14 @@ sub downloadruleset ($) { # Break loop. last; + # Check if the server responds with 304 (Not Modified). + } elsif ($response->code == 304) { + # Log to syslog. + &_log_to_syslog("Ruleset is up-to-date, no update required."); + + # Nothing to be done, the ruleset is up-to-date. + return; + # Check if we ran out of download re-tries. } elsif ($dl_attempt eq $max_dl_attempts) { # Obtain error. @@ -406,6 +430,10 @@ sub downloadruleset ($) { # Get the remote size of the downloaded file. my $remote_filesize = $headers->content_length; + # Get the timestamp from header, when the file has been modified the + # last time. + my $last_modified = $headers->last_modified; + # Load perl stat module. use File::stat; @@ -428,9 +456,6 @@ sub downloadruleset ($) { return 1; } - # Genarate and assign file name and path to store the downloaded rules file. - my $dl_rulesfile = &_get_dl_rulesfile($provider); - # Check if a file name could be obtained. unless ($dl_rulesfile) { # Log error message. @@ -449,6 +474,13 @@ sub downloadruleset ($) { # Overwrite the may existing rulefile or tarball with the downloaded one. move("$tmpfile", "$dl_rulesfile"); + # Check if the server respond contained a last_modified value. + if ($last_modified) { + # Assign the last modified timestamp from server as mtime to the + # rules file. + utime(time(), "$last_modified", "$dl_rulesfile"); + } + # Delete temporary file. unlink("$tmpfile");