From patchwork Wed Mar 23 04:04:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Schantl X-Patchwork-Id: 5389 Return-Path: Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384 client-signature ECDSA (P-384) client-digest SHA384) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by web04.haj.ipfire.org (Postfix) with ESMTPS id 4KNZWR5KNpz3wtb for ; Wed, 23 Mar 2022 04:05:07 +0000 (UTC) Received: from mail02.haj.ipfire.org (mail02.haj.ipfire.org [172.28.1.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail02.haj.ipfire.org", Issuer "R3" (verified OK)) by mail01.ipfire.org (Postfix) with ESMTPS id 4KNZWM0yJDz5HR; Wed, 23 Mar 2022 04:05:03 +0000 (UTC) Received: from mail02.haj.ipfire.org (localhost [127.0.0.1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4KNZWL2cVFz2yyB; Wed, 23 Mar 2022 04:05:02 +0000 (UTC) Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4KNZWJ158Pz2xqt for ; Wed, 23 Mar 2022 04:05:00 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by mail01.ipfire.org (Postfix) with ESMTPSA id 4KNZWH3GSSzLX; Wed, 23 Mar 2022 04:04:59 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003ed25519; t=1648008299; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=LMyu3ifpewF6OzHYmJun+0YYw4qUc98N6c06d9hnskY=; b=SZBqy4dbU/du11NvUh2UWpKVDkOTkAXp+B5vwGKD6vGVZGvDBr9vt7QHQICq75UDj4d0oA M8XybvU5bJZr11DQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003rsa; t=1648008299; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=LMyu3ifpewF6OzHYmJun+0YYw4qUc98N6c06d9hnskY=; b=ouUT+pdDPxYIkgmG3hSNEh2xT/AYoA+lVcPmwN2FsduxAll+hX0onuggJjxzupO3CiCKwy vZqrOIaWxkKeGWXaph+GUc7tn2Sno1k7Xuj/pW2gnNbRHZUNKTrC/YJ81xr8vLWnXKqEAP DnTWg/WQySXK9yaOnQs+h6w1m0R9BLwNzm9L4WgNxI64Qg8PZCWr/uDQ1vldN356ONTcTE vzA/TWvGLMiRm507i7mi1EoujmCzW1WYM6fdLEhptjJEb1d7ac0iyaA5/zRzJreKCovrBR ZX0joTiR/TmXD4HtLOwlBhq2PEsucRkm2U84bFmgQ+7SdM3CDHgEysk5nR8x2Q== From: Stefan Schantl To: development@lists.ipfire.org Subject: [PATCH 1/5] ids-functions.pl: Drop downloader code for sourcefire based ruleset. Date: Wed, 23 Mar 2022 05:04:48 +0100 Message-Id: <20220323040452.2609-1-stefan.schantl@ipfire.org> MIME-Version: 1.0 X-BeenThere: development@lists.ipfire.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: IPFire development talk List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: development-bounces@lists.ipfire.org Sender: "Development" Even if the servers do not support HEAD requests, the remote filesize (content_length) can be obtained from the connection headers. This generic method works for all servers and therefore we do not need the code for handle sourcefire servers in a different way anymore. Signed-off-by: Stefan Schantl --- config/cfgroot/ids-functions.pl | 43 +++++---------------------------- 1 file changed, 6 insertions(+), 37 deletions(-) diff --git a/config/cfgroot/ids-functions.pl b/config/cfgroot/ids-functions.pl index 94dccc8ae..eb276030b 100644 --- a/config/cfgroot/ids-functions.pl +++ b/config/cfgroot/ids-functions.pl @@ -354,43 +354,6 @@ sub downloadruleset ($) { return 1; } - # Variable to store the filesize of the remote object. - my $remote_filesize; - - # The sourcfire (snort rules) does not allow to send "HEAD" requests, so skip this check - # for this webserver. - # - # Check if the ruleset source contains "snort.org". - unless ($url =~ /\.snort\.org/) { - # Pass the requrested url to the downloader. - my $request = HTTP::Request->new(HEAD => $url); - - # Accept the html header. - $request->header('Accept' => 'text/html'); - - # Perform the request and fetch the html header. - my $response = $downloader->request($request); - - # Check if there was any error. - unless ($response->is_success) { - # Obtain error. - my $error = $response->status_line(); - - # Log error message. - &_log_to_syslog("Unable to download the ruleset. \($error\)"); - - # Return "1" - false. - return 1; - } - - # Assign the fetched header object. - my $header = $response->headers(); - - # Grab the remote file size from the object and store it in the - # variable. - $remote_filesize = $header->content_length; - } - # Load perl module to deal with temporary files. use File::Temp; @@ -416,6 +379,12 @@ sub downloadruleset ($) { return 1; } + # Obtain the connection headers. + my $headers = $response->headers; + + # Get the remote size of the downloaded file. + my $remote_filesize = $headers->content_length; + # Load perl stat module. use File::stat; From patchwork Wed Mar 23 04:04:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Schantl X-Patchwork-Id: 5392 Return-Path: Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by web04.haj.ipfire.org (Postfix) with ESMTPS id 4KNZWT0JFjz3xqg for ; Wed, 23 Mar 2022 04:05:09 +0000 (UTC) Received: from mail02.haj.ipfire.org (mail02.haj.ipfire.org [172.28.1.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail02.haj.ipfire.org", Issuer "R3" (verified OK)) by mail01.ipfire.org (Postfix) with ESMTPS id 4KNZWM1j6qz5Qb; Wed, 23 Mar 2022 04:05:03 +0000 (UTC) Received: from mail02.haj.ipfire.org (localhost [127.0.0.1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4KNZWL3Jn5z300y; Wed, 23 Mar 2022 04:05:02 +0000 (UTC) Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4KNZWJ3VVfz2xqt for ; Wed, 23 Mar 2022 04:05:00 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by mail01.ipfire.org (Postfix) with ESMTPSA id 4KNZWJ0g7yz1PB; Wed, 23 Mar 2022 04:05:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003rsa; t=1648008300; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JeCV7yssrEZvGRF+5cqew9vwgOGyRd2uJeYujMJjUyQ=; b=mAZ/7HN8zntBEfG/9LeNKS0r/+m5XhzfrFP5ZhyyWVdCYa/7h5bo66v2qkYMejNu8fhFlN Nlk4RpkijUDuIFTWCVa8UBufn4xoSWHitAsEhuJeEW8IvpH6PkDwRWIiL2IJ0fYyVxli5A +LfZpojAuwiN33qva7Pmwqw/MyqOR8C2adwoOChknBkU9rFVZVeHsTaQDpmLThpXjQMmQ+ ikEi2TIpwvFZbnM60295f5zvUo1DmWDYv5PecNK90WVdUQccuiy/tTOm/h4P9yI+NuzbcP 0w3W28E0hA2g8SWyfjquWzPyWhB4DyxMAd9oyog6K7XsEs0NmXYIEkGh3GLH4g== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003ed25519; t=1648008300; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JeCV7yssrEZvGRF+5cqew9vwgOGyRd2uJeYujMJjUyQ=; b=9yW8jvkrF/bTd3Rn+ZmOC5EeOOfX2XHReT8zv3Yo7UQ5mzU2YjudIeDzFI0SDuAfD9ilv9 wlWKvOrhpLJKb1Dw== From: Stefan Schantl To: development@lists.ipfire.org Subject: [PATCH 2/5] ids-functions.pl: Allow "5" download attempts for each provider before fail. Date: Wed, 23 Mar 2022 05:04:49 +0100 Message-Id: <20220323040452.2609-2-stefan.schantl@ipfire.org> In-Reply-To: <20220323040452.2609-1-stefan.schantl@ipfire.org> References: <20220323040452.2609-1-stefan.schantl@ipfire.org> MIME-Version: 1.0 X-BeenThere: development@lists.ipfire.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: IPFire development talk List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: development-bounces@lists.ipfire.org Sender: "Development" Signed-off-by: Stefan Schantl --- config/cfgroot/ids-functions.pl | 38 ++++++++++++++++++++++++--------- 1 file changed, 28 insertions(+), 10 deletions(-) diff --git a/config/cfgroot/ids-functions.pl b/config/cfgroot/ids-functions.pl index eb276030b..c8bc52b1b 100644 --- a/config/cfgroot/ids-functions.pl +++ b/config/cfgroot/ids-functions.pl @@ -256,6 +256,10 @@ sub downloadruleset ($) { # If no provider is given default to "all". $provider //= 'all'; + # The amount of download attempts before giving up and + # logging an error. + my $max_dl_attempts = 5; + # Hash to store the providers and access id's, for which rules should be downloaded. my %sheduled_providers = (); @@ -364,19 +368,33 @@ sub downloadruleset ($) { # Pass the requested url to the downloader. my $request = HTTP::Request->new(GET => $url); - # Perform the request and save the output into the tmpfile. - my $response = $downloader->request($request, $tmpfile); + my $dl_attempt = 1; + my $response; - # Check if there was any error. - unless ($response->is_success) { - # Obtain error. - my $error = $response->content; + # Download and retry on failure. + while ($dl_attempt <= $max_dl_attempts) { + # Perform the request and save the output into the tmpfile. + $response = $downloader->request($request, $tmpfile); - # Log error message. - &_log_to_syslog("Unable to download the ruleset. \($error\)"); + # Check if the download was successfull. + if($response->is_success) { + # Break loop. + last; - # Return "1" - false. - return 1; + # Check if we ran out of download re-tries. + } elsif ($dl_attempt eq $max_dl_attempts) { + # Obtain error. + my $error = $response->content; + + # Log error message. + &_log_to_syslog("Unable to download the ruleset. \($error\)"); + + # Return "1" - false. + return 1; + } + + # Increase download attempt counter. + $dl_attempt++; } # Obtain the connection headers. From patchwork Wed Mar 23 04:04:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Schantl X-Patchwork-Id: 5390 Return-Path: Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by web04.haj.ipfire.org (Postfix) with ESMTPS id 4KNZWR5NCnz3xq8 for ; Wed, 23 Mar 2022 04:05:07 +0000 (UTC) Received: from mail02.haj.ipfire.org (mail02.haj.ipfire.org [172.28.1.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail02.haj.ipfire.org", Issuer "R3" (verified OK)) by mail01.ipfire.org (Postfix) with ESMTPS id 4KNZWM3D1Dz5S7; Wed, 23 Mar 2022 04:05:03 +0000 (UTC) Received: from mail02.haj.ipfire.org (localhost [127.0.0.1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4KNZWL3vfVz2yyR; Wed, 23 Mar 2022 04:05:02 +0000 (UTC) Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4KNZWJ6hBFz2xqt for ; Wed, 23 Mar 2022 04:05:00 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by mail01.ipfire.org (Postfix) with ESMTPSA id 4KNZWJ3WGRz1hX; Wed, 23 Mar 2022 04:05:00 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003ed25519; t=1648008300; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FNklpXOAyXg6LctDs9JNxASxGsaRrgKxAitoWULYZhY=; b=b6F2pRF4sa0NryX+GsMTKlqRC9rS7DM8OXGvk0TtzNY1HhtcNA/sV5senJugNUscU5v8N5 mBzv3tM68GJCYuBA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003rsa; t=1648008300; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FNklpXOAyXg6LctDs9JNxASxGsaRrgKxAitoWULYZhY=; b=FiS2wiPIxh8iV1A/963BVAltf74a6JAT7TGOlJ6LPikYLlUFM8zaFylmWYgZ2OAz68dmJ9 3zonGNjugsjWpCOQnJwhM5fBjULpaHL30/B+2O+6dvp3dy6GVMPb2T4RnBoc3F6ShhXNlT eL26m8iAbsQH8tSpBzfqcAzEdJ53vTr5sPG/IR5e5qJKc9zpw405XjemvYG8rpx3rj/Qxe 200CVxQlf0azR2e4P2IFfzAwKsOTL28h2/MVY7WyByOc3jsISo3KAykzhJkhhjVi3rcJVT Zbqz3m+n7Iv8fLUyPuoOdHSDDVS5QU5WY+kS25kytbXIj54w9qc50wMwsMhN6w== From: Stefan Schantl To: development@lists.ipfire.org Subject: [PATCH 3/5] ids-functions.pl: Remove temporary file, if the download failed. Date: Wed, 23 Mar 2022 05:04:50 +0100 Message-Id: <20220323040452.2609-3-stefan.schantl@ipfire.org> In-Reply-To: <20220323040452.2609-1-stefan.schantl@ipfire.org> References: <20220323040452.2609-1-stefan.schantl@ipfire.org> MIME-Version: 1.0 X-BeenThere: development@lists.ipfire.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: IPFire development talk List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: development-bounces@lists.ipfire.org Sender: "Development" Signed-off-by: Stefan Schantl --- config/cfgroot/ids-functions.pl | 3 +++ 1 file changed, 3 insertions(+) diff --git a/config/cfgroot/ids-functions.pl b/config/cfgroot/ids-functions.pl index c8bc52b1b..dfbeb1a7d 100644 --- a/config/cfgroot/ids-functions.pl +++ b/config/cfgroot/ids-functions.pl @@ -393,6 +393,9 @@ sub downloadruleset ($) { return 1; } + # Remove temporary file, if one exists. + unlink("$tmpfile") if (-e "$tmpfile"); + # Increase download attempt counter. $dl_attempt++; } From patchwork Wed Mar 23 04:04:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Schantl X-Patchwork-Id: 5391 Return-Path: Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by web04.haj.ipfire.org (Postfix) with ESMTPS id 4KNZWS1L1Tz3xqd for ; Wed, 23 Mar 2022 04:05:08 +0000 (UTC) Received: from mail02.haj.ipfire.org (mail02.haj.ipfire.org [172.28.1.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail02.haj.ipfire.org", Issuer "R3" (verified OK)) by mail01.ipfire.org (Postfix) with ESMTPS id 4KNZWN0Ykyz5WB; Wed, 23 Mar 2022 04:05:04 +0000 (UTC) Received: from mail02.haj.ipfire.org (localhost [127.0.0.1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4KNZWM4pM2z3009; Wed, 23 Mar 2022 04:05:03 +0000 (UTC) Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4KNZWK2Nqnz2yxF for ; Wed, 23 Mar 2022 04:05:01 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by mail01.ipfire.org (Postfix) with ESMTPSA id 4KNZWJ6cS0z5HR; Wed, 23 Mar 2022 04:05:00 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003ed25519; t=1648008301; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=uMxCrqyDrzrfCF/FVN1UL4sgIdLUK1vkwKaR7OmvQ4k=; b=mdWarz3jetrnze/V7f2emJmCS8rG6Rr5Uksi92nvwfwdNGQ+ILgDpFXTyJ3rkJXjo6Tijq RWhZSaoyOkM2eZAg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003rsa; t=1648008301; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=uMxCrqyDrzrfCF/FVN1UL4sgIdLUK1vkwKaR7OmvQ4k=; b=nNIwxTP7iGScn3CELR3YXprAmQ4zs4Iz2zlfuK7K5ujVLygTlQLymRoFGgAGgSyH0MTh9D 1r1fpZGzF9IK4PWMmupNLELaNfBkRPv9GZN/N4fJs5N0ySAaPa3I0jfrHBUNvqrO6INB88 gwbmO26hVF8cx0ELrB+SEJuhkU5YxEv+C8v0oLUAmH/C8I+azySjh/2zRFuwUyGCeFhy4g SHnVjyi/b26xHK4XL1TfMmtL2jJgaS0JWQfprHHIhW3mnNX6gnHbHCXs7t5e5+3q6AtKpm 6NlLVMkZSyVCr4l0YOLcyK6xpppiQtAhE/an1uNZ7rFBtNGl9otnxt7k5cmJ/Q== From: Stefan Schantl To: development@lists.ipfire.org Subject: [PATCH 4/5] ids-functions.pl: Use If-Modified-Since header to reduce file downloads. Date: Wed, 23 Mar 2022 05:04:51 +0100 Message-Id: <20220323040452.2609-4-stefan.schantl@ipfire.org> In-Reply-To: <20220323040452.2609-1-stefan.schantl@ipfire.org> References: <20220323040452.2609-1-stefan.schantl@ipfire.org> MIME-Version: 1.0 X-BeenThere: development@lists.ipfire.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: IPFire development talk List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: development-bounces@lists.ipfire.org Sender: "Development" When using the "If-Modified-Since" header, the server can be requested if a modified version of the file can be served. In case that is true, the file will be sent and stored by the downloader function. If the file has not been touched since the last time, the server will respond with the code "304" (Not modified). This tells us, that the current stored file is the latest one (still up-to-date) and we safely can skip the download attempt for this provider. Signed-off-by: Stefan Schantl --- config/cfgroot/ids-functions.pl | 38 ++++++++++++++++++++++++++++++--- 1 file changed, 35 insertions(+), 3 deletions(-) diff --git a/config/cfgroot/ids-functions.pl b/config/cfgroot/ids-functions.pl index dfbeb1a7d..d7df41dd1 100644 --- a/config/cfgroot/ids-functions.pl +++ b/config/cfgroot/ids-functions.pl @@ -365,9 +365,25 @@ sub downloadruleset ($) { my $tmp = File::Temp->new( SUFFIX => ".tmp", DIR => "/var/tmp/", UNLINK => 0 ); my $tmpfile = $tmp->filename(); + # Genarate and assign file name and path to store the downloaded rules file. + my $dl_rulesfile = &_get_dl_rulesfile($provider); + + # Load perl module to deal with file atributes. + use File::stat; + + # Get the mtime of the rulesfile if it exists. + my $mtime = (stat($dl_rulesfile)->mtime) if (-f $dl_rulesfile); + + # Convert the mtime into gmtime format. + my $gmtime = gmtime($mtime || 0); + # Pass the requested url to the downloader. my $request = HTTP::Request->new(GET => $url); + # Add the If-Modified-Since header to the request, containing the omited and converted + # mtime of the downloaded rules file, if one is present. + $request->header( 'If-Modified-Since' => "$gmtime" ); + my $dl_attempt = 1; my $response; @@ -381,6 +397,14 @@ sub downloadruleset ($) { # Break loop. last; + # Check if the server responds with 304 (Not Modified). + } elsif ($response->code == 304) { + # Log to syslog. + &_log_to_syslog("Ruleset is up-to-date, no update required."); + + # Nothing to be done, the ruleset is up-to-date. + return; + # Check if we ran out of download re-tries. } elsif ($dl_attempt eq $max_dl_attempts) { # Obtain error. @@ -406,6 +430,10 @@ sub downloadruleset ($) { # Get the remote size of the downloaded file. my $remote_filesize = $headers->content_length; + # Get the timestamp from header, when the file has been modified the + # last time. + my $last_modified = $headers->last_modified; + # Load perl stat module. use File::stat; @@ -428,9 +456,6 @@ sub downloadruleset ($) { return 1; } - # Genarate and assign file name and path to store the downloaded rules file. - my $dl_rulesfile = &_get_dl_rulesfile($provider); - # Check if a file name could be obtained. unless ($dl_rulesfile) { # Log error message. @@ -449,6 +474,13 @@ sub downloadruleset ($) { # Overwrite the may existing rulefile or tarball with the downloaded one. move("$tmpfile", "$dl_rulesfile"); + # Check if the server respond contained a last_modified value. + if ($last_modified) { + # Assign the last modified timestamp from server as mtime to the + # rules file. + utime(time(), "$last_modified", "$dl_rulesfile"); + } + # Delete temporary file. unlink("$tmpfile"); From patchwork Wed Mar 23 04:04:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Schantl X-Patchwork-Id: 5393 Return-Path: Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by web04.haj.ipfire.org (Postfix) with ESMTPS id 4KNZWY2DLwz3wtb for ; Wed, 23 Mar 2022 04:05:13 +0000 (UTC) Received: from mail02.haj.ipfire.org (mail02.haj.ipfire.org [172.28.1.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail02.haj.ipfire.org", Issuer "R3" (verified OK)) by mail01.ipfire.org (Postfix) with ESMTPS id 4KNZWN3Vggz5Vj; Wed, 23 Mar 2022 04:05:04 +0000 (UTC) Received: from mail02.haj.ipfire.org (localhost [127.0.0.1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4KNZWM5k3Wz2yyB; Wed, 23 Mar 2022 04:05:03 +0000 (UTC) Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4KNZWK58D0z2yxF for ; Wed, 23 Mar 2022 04:05:01 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by mail01.ipfire.org (Postfix) with ESMTPSA id 4KNZWK2LQkzLX; Wed, 23 Mar 2022 04:05:01 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003ed25519; t=1648008301; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Lp8lyILpJuR/PhDq086jINK2FOinJPqIfslugJpdni8=; b=YRBJHr8qWoz4e1hgdJ1Xr2eSw7hplrp8V8Jd2vehMVMOLiLOUaw5PLbzWHLHWe+/gvUYBw d5wMhylzTtzLYoAQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003rsa; t=1648008301; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Lp8lyILpJuR/PhDq086jINK2FOinJPqIfslugJpdni8=; b=HBFuJEzv5ywL0G11ES4h748NTxcfbFzUYypevKckffbZBdLN/Y37tHfkWEwa7UaqYnnHXU SC3NJck4UVhpgdWK5fngUnwqEyH2NVQ2fP3hVnfqJ6CBFwBSE+7bGqZEQbnGZMFYnTkD5t WmfRvYNBmNMbDZRPY3llm7aSWH/2JG8p9W6VRzabEB6ogxZXeKPMMZoiso8it46QA8OE1y 6RSPcgfNJXKsEcGk21W+fqhMP89UVEch7LxpOLbfuplHBT334ylyJyd76e0vtWbZU7r6Co mcP8qNvYHVrxNGkZEs4jmC8Y/pJjvcZ83Ns9YD/BJwQhPjqMHIP2v/j93vBdTQ== From: Stefan Schantl To: development@lists.ipfire.org Subject: [PATCH 5/5] ids-functions.pl: Do not longer call any log message as "ERROR". Date: Wed, 23 Mar 2022 05:04:52 +0100 Message-Id: <20220323040452.2609-5-stefan.schantl@ipfire.org> In-Reply-To: <20220323040452.2609-1-stefan.schantl@ipfire.org> References: <20220323040452.2609-1-stefan.schantl@ipfire.org> MIME-Version: 1.0 X-BeenThere: development@lists.ipfire.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: IPFire development talk List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: development-bounces@lists.ipfire.org Sender: "Development" Fixes #12805. Signed-off-by: Stefan Schantl --- config/cfgroot/ids-functions.pl | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/config/cfgroot/ids-functions.pl b/config/cfgroot/ids-functions.pl index d7df41dd1..9eb375bc9 100644 --- a/config/cfgroot/ids-functions.pl +++ b/config/cfgroot/ids-functions.pl @@ -226,7 +226,7 @@ sub checkdiskspace () { # Check if the available disk space is more than 300MB. if ($available < 300) { # Log error to syslog. - &_log_to_syslog("Not enough free disk space on /var. Only $available MB from 300 MB available."); + &_log_to_syslog(" Not enough free disk space on /var. Only $available MB from 300 MB available."); # Exit function and return "1" - False. return 1; @@ -270,7 +270,7 @@ sub downloadruleset ($) { # Check if a ruleset has been configured. unless(%used_providers) { # Log that no ruleset has been configured and abort. - &_log_to_syslog("No ruleset provider has been configured."); + &_log_to_syslog(" No ruleset provider has been configured."); # Return "1". return 1; @@ -333,7 +333,7 @@ sub downloadruleset ($) { # Loop through the hash of sheduled providers. foreach my $provider ( keys %sheduled_providers) { # Log download/update of the ruleset. - &_log_to_syslog("Downloading ruleset for provider: $provider."); + &_log_to_syslog(" Downloading ruleset for provider: $provider."); # Grab the download url for the provider. my $url = $IDS::Ruleset::Providers{$provider}{'dl_url'}; @@ -354,7 +354,7 @@ sub downloadruleset ($) { # Abort if no url could be determined for the provider. unless ($url) { # Log error and abort. - &_log_to_syslog("Unable to gather a download URL for the selected ruleset provider."); + &_log_to_syslog(" Unable to gather a download URL for the selected ruleset provider."); return 1; } @@ -400,7 +400,7 @@ sub downloadruleset ($) { # Check if the server responds with 304 (Not Modified). } elsif ($response->code == 304) { # Log to syslog. - &_log_to_syslog("Ruleset is up-to-date, no update required."); + &_log_to_syslog(" Ruleset is up-to-date, no update required."); # Nothing to be done, the ruleset is up-to-date. return; @@ -411,7 +411,7 @@ sub downloadruleset ($) { my $error = $response->content; # Log error message. - &_log_to_syslog("Unable to download the ruleset. \($error\)"); + &_log_to_syslog(" Unable to download the ruleset. \($error\)"); # Return "1" - false. return 1; @@ -446,8 +446,8 @@ sub downloadruleset ($) { # Check if both file sizes match. if (($remote_filesize) && ($remote_filesize ne $local_filesize)) { # Log error message. - &_log_to_syslog("Unable to completely download the ruleset. "); - &_log_to_syslog("Only got $local_filesize Bytes instead of $remote_filesize Bytes. "); + &_log_to_syslog(" Unable to completely download the ruleset. "); + &_log_to_syslog(" Only got $local_filesize Bytes instead of $remote_filesize Bytes. "); # Delete temporary file. unlink("$tmpfile"); @@ -459,7 +459,7 @@ sub downloadruleset ($) { # Check if a file name could be obtained. unless ($dl_rulesfile) { # Log error message. - &_log_to_syslog("Unable to store the downloaded rules file. "); + &_log_to_syslog(" Unable to store the downloaded rules file. "); # Delete downloaded temporary file. unlink("$tmpfile"); @@ -518,7 +518,7 @@ sub extractruleset ($) { # Check if the file exists. unless (-f $tarball) { - &_log_to_syslog("Could not find ruleset file: $tarball"); + &_log_to_syslog(" Could not find ruleset file: $tarball"); # Return nothing. return; @@ -897,7 +897,7 @@ sub _log_to_syslog ($) { # The syslog function works best with an array based input, # so generate one before passing the message details to syslog. - my @syslog = ("ERR", " $message"); + my @syslog = ("ERR", "$message"); # Establish the connection to the syslog service. openlog('oinkmaster', 'cons,pid', 'user');