From patchwork Wed Mar 23 04:04:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Schantl X-Patchwork-Id: 5391 Return-Path: Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by web04.haj.ipfire.org (Postfix) with ESMTPS id 4KNZWS1L1Tz3xqd for ; Wed, 23 Mar 2022 04:05:08 +0000 (UTC) Received: from mail02.haj.ipfire.org (mail02.haj.ipfire.org [172.28.1.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail02.haj.ipfire.org", Issuer "R3" (verified OK)) by mail01.ipfire.org (Postfix) with ESMTPS id 4KNZWN0Ykyz5WB; Wed, 23 Mar 2022 04:05:04 +0000 (UTC) Received: from mail02.haj.ipfire.org (localhost [127.0.0.1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4KNZWM4pM2z3009; Wed, 23 Mar 2022 04:05:03 +0000 (UTC) Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4KNZWK2Nqnz2yxF for ; Wed, 23 Mar 2022 04:05:01 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by mail01.ipfire.org (Postfix) with ESMTPSA id 4KNZWJ6cS0z5HR; Wed, 23 Mar 2022 04:05:00 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003ed25519; t=1648008301; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=uMxCrqyDrzrfCF/FVN1UL4sgIdLUK1vkwKaR7OmvQ4k=; b=mdWarz3jetrnze/V7f2emJmCS8rG6Rr5Uksi92nvwfwdNGQ+ILgDpFXTyJ3rkJXjo6Tijq RWhZSaoyOkM2eZAg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003rsa; t=1648008301; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=uMxCrqyDrzrfCF/FVN1UL4sgIdLUK1vkwKaR7OmvQ4k=; b=nNIwxTP7iGScn3CELR3YXprAmQ4zs4Iz2zlfuK7K5ujVLygTlQLymRoFGgAGgSyH0MTh9D 1r1fpZGzF9IK4PWMmupNLELaNfBkRPv9GZN/N4fJs5N0ySAaPa3I0jfrHBUNvqrO6INB88 gwbmO26hVF8cx0ELrB+SEJuhkU5YxEv+C8v0oLUAmH/C8I+azySjh/2zRFuwUyGCeFhy4g SHnVjyi/b26xHK4XL1TfMmtL2jJgaS0JWQfprHHIhW3mnNX6gnHbHCXs7t5e5+3q6AtKpm 6NlLVMkZSyVCr4l0YOLcyK6xpppiQtAhE/an1uNZ7rFBtNGl9otnxt7k5cmJ/Q== From: Stefan Schantl To: development@lists.ipfire.org Subject: [PATCH 4/5] ids-functions.pl: Use If-Modified-Since header to reduce file downloads. Date: Wed, 23 Mar 2022 05:04:51 +0100 Message-Id: <20220323040452.2609-4-stefan.schantl@ipfire.org> In-Reply-To: <20220323040452.2609-1-stefan.schantl@ipfire.org> References: <20220323040452.2609-1-stefan.schantl@ipfire.org> MIME-Version: 1.0 X-BeenThere: development@lists.ipfire.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: IPFire development talk List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: development-bounces@lists.ipfire.org Sender: "Development" When using the "If-Modified-Since" header, the server can be requested if a modified version of the file can be served. In case that is true, the file will be sent and stored by the downloader function. If the file has not been touched since the last time, the server will respond with the code "304" (Not modified). This tells us, that the current stored file is the latest one (still up-to-date) and we safely can skip the download attempt for this provider. Signed-off-by: Stefan Schantl --- config/cfgroot/ids-functions.pl | 38 ++++++++++++++++++++++++++++++--- 1 file changed, 35 insertions(+), 3 deletions(-) diff --git a/config/cfgroot/ids-functions.pl b/config/cfgroot/ids-functions.pl index dfbeb1a7d..d7df41dd1 100644 --- a/config/cfgroot/ids-functions.pl +++ b/config/cfgroot/ids-functions.pl @@ -365,9 +365,25 @@ sub downloadruleset ($) { my $tmp = File::Temp->new( SUFFIX => ".tmp", DIR => "/var/tmp/", UNLINK => 0 ); my $tmpfile = $tmp->filename(); + # Genarate and assign file name and path to store the downloaded rules file. + my $dl_rulesfile = &_get_dl_rulesfile($provider); + + # Load perl module to deal with file atributes. + use File::stat; + + # Get the mtime of the rulesfile if it exists. + my $mtime = (stat($dl_rulesfile)->mtime) if (-f $dl_rulesfile); + + # Convert the mtime into gmtime format. + my $gmtime = gmtime($mtime || 0); + # Pass the requested url to the downloader. my $request = HTTP::Request->new(GET => $url); + # Add the If-Modified-Since header to the request, containing the omited and converted + # mtime of the downloaded rules file, if one is present. + $request->header( 'If-Modified-Since' => "$gmtime" ); + my $dl_attempt = 1; my $response; @@ -381,6 +397,14 @@ sub downloadruleset ($) { # Break loop. last; + # Check if the server responds with 304 (Not Modified). + } elsif ($response->code == 304) { + # Log to syslog. + &_log_to_syslog("Ruleset is up-to-date, no update required."); + + # Nothing to be done, the ruleset is up-to-date. + return; + # Check if we ran out of download re-tries. } elsif ($dl_attempt eq $max_dl_attempts) { # Obtain error. @@ -406,6 +430,10 @@ sub downloadruleset ($) { # Get the remote size of the downloaded file. my $remote_filesize = $headers->content_length; + # Get the timestamp from header, when the file has been modified the + # last time. + my $last_modified = $headers->last_modified; + # Load perl stat module. use File::stat; @@ -428,9 +456,6 @@ sub downloadruleset ($) { return 1; } - # Genarate and assign file name and path to store the downloaded rules file. - my $dl_rulesfile = &_get_dl_rulesfile($provider); - # Check if a file name could be obtained. unless ($dl_rulesfile) { # Log error message. @@ -449,6 +474,13 @@ sub downloadruleset ($) { # Overwrite the may existing rulefile or tarball with the downloaded one. move("$tmpfile", "$dl_rulesfile"); + # Check if the server respond contained a last_modified value. + if ($last_modified) { + # Assign the last modified timestamp from server as mtime to the + # rules file. + utime(time(), "$last_modified", "$dl_rulesfile"); + } + # Delete temporary file. unlink("$tmpfile");