From patchwork Tue Mar 22 19:40:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Schantl X-Patchwork-Id: 5385 Return-Path: Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by web04.haj.ipfire.org (Postfix) with ESMTPS id 4KNML10BJ3z3xqb for ; Tue, 22 Mar 2022 19:41:13 +0000 (UTC) Received: from mail02.haj.ipfire.org (mail02.haj.ipfire.org [172.28.1.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail02.haj.ipfire.org", Issuer "R3" (verified OK)) by mail01.ipfire.org (Postfix) with ESMTPS id 4KNMKy191nz4RG; Tue, 22 Mar 2022 19:41:10 +0000 (UTC) Received: from mail02.haj.ipfire.org (localhost [127.0.0.1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4KNMKx316Mz2ywN; Tue, 22 Mar 2022 19:41:09 +0000 (UTC) Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4KNMKv20gXz2xWt for ; Tue, 22 Mar 2022 19:41:07 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by mail01.ipfire.org (Postfix) with ESMTPSA id 4KNMKs4s33zLX; Tue, 22 Mar 2022 19:41:05 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003ed25519; t=1647978066; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=LMyu3ifpewF6OzHYmJun+0YYw4qUc98N6c06d9hnskY=; b=L00seMPl1UtwrX9bGUdw9yd1zClvOv+iM1340ksH0v+/0P9i9qaAqbvEXuazTdjoY2Eqfx dOus57nVf8sor6Dw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003rsa; t=1647978066; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=LMyu3ifpewF6OzHYmJun+0YYw4qUc98N6c06d9hnskY=; b=HSogsi3K0DFYRp1VehjDGXdq2gbp+ovGkx2laS72qhFqEIWXI+n1KUywNDx/Y+EU7QRKjN NmnnhDQ96djSt2OhuQnt1cd5iKE/4X9eGNIQNvoTZwby2NrczRl9XwVQgfXv7rPDz80ahl pi0CYZolAI8eMiHJP0VM8goT/jW7do0GIJm/NICcMLyM7pXsTt2CUZw3cFCwF7CHaYV8L8 n3U7FtJWBGoV+atvvDjMVKPe+/0USBJz4ydACygBq9XX2IKgExqM2xjEHgX8tzAtienNYI uuFf8lgFYKsSjYNIYyug3trPE+hFV+5kTi3fggcjmoVLZ0iIe/H4CXmJhnA1RQ== From: Stefan Schantl To: development@lists.ipfire.org Subject: [PATCH 1/5] ids-functions.pl: Drop downloader code for sourcefire based ruleset. Date: Tue, 22 Mar 2022 20:40:55 +0100 Message-Id: <20220322194059.3030-1-stefan.schantl@ipfire.org> MIME-Version: 1.0 X-BeenThere: development@lists.ipfire.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: IPFire development talk List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: development-bounces@lists.ipfire.org Sender: "Development" Even if the servers do not support HEAD requests, the remote filesize (content_length) can be obtained from the connection headers. This generic method works for all servers and therefore we do not need the code for handle sourcefire servers in a different way anymore. Signed-off-by: Stefan Schantl --- config/cfgroot/ids-functions.pl | 43 +++++---------------------------- 1 file changed, 6 insertions(+), 37 deletions(-) diff --git a/config/cfgroot/ids-functions.pl b/config/cfgroot/ids-functions.pl index 94dccc8ae..eb276030b 100644 --- a/config/cfgroot/ids-functions.pl +++ b/config/cfgroot/ids-functions.pl @@ -354,43 +354,6 @@ sub downloadruleset ($) { return 1; } - # Variable to store the filesize of the remote object. - my $remote_filesize; - - # The sourcfire (snort rules) does not allow to send "HEAD" requests, so skip this check - # for this webserver. - # - # Check if the ruleset source contains "snort.org". - unless ($url =~ /\.snort\.org/) { - # Pass the requrested url to the downloader. - my $request = HTTP::Request->new(HEAD => $url); - - # Accept the html header. - $request->header('Accept' => 'text/html'); - - # Perform the request and fetch the html header. - my $response = $downloader->request($request); - - # Check if there was any error. - unless ($response->is_success) { - # Obtain error. - my $error = $response->status_line(); - - # Log error message. - &_log_to_syslog("Unable to download the ruleset. \($error\)"); - - # Return "1" - false. - return 1; - } - - # Assign the fetched header object. - my $header = $response->headers(); - - # Grab the remote file size from the object and store it in the - # variable. - $remote_filesize = $header->content_length; - } - # Load perl module to deal with temporary files. use File::Temp; @@ -416,6 +379,12 @@ sub downloadruleset ($) { return 1; } + # Obtain the connection headers. + my $headers = $response->headers; + + # Get the remote size of the downloaded file. + my $remote_filesize = $headers->content_length; + # Load perl stat module. use File::stat; From patchwork Tue Mar 22 19:40:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Schantl X-Patchwork-Id: 5384 Return-Path: Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384 client-signature ECDSA (P-384) client-digest SHA384) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by web04.haj.ipfire.org (Postfix) with ESMTPS id 4KNML04B4Tz3xlr for ; Tue, 22 Mar 2022 19:41:12 +0000 (UTC) Received: from mail02.haj.ipfire.org (mail02.haj.ipfire.org [172.28.1.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail02.haj.ipfire.org", Issuer "R3" (verified OK)) by mail01.ipfire.org (Postfix) with ESMTPS id 4KNMKx3lb1zLX; Tue, 22 Mar 2022 19:41:09 +0000 (UTC) Received: from mail02.haj.ipfire.org (localhost [127.0.0.1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4KNMKx2KTCz2yt6; Tue, 22 Mar 2022 19:41:09 +0000 (UTC) Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384 client-signature ECDSA (P-384) client-digest SHA384) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4KNMKv1zYdz2xPW for ; Tue, 22 Mar 2022 19:41:07 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by mail01.ipfire.org (Postfix) with ESMTPSA id 4KNMKt4QgjzxJ; Tue, 22 Mar 2022 19:41:06 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003ed25519; t=1647978066; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JeCV7yssrEZvGRF+5cqew9vwgOGyRd2uJeYujMJjUyQ=; b=DzeNEZORaq3mf8C9HV9SeLYNgZB1HWeSZYSRTCpDwYWx0/oQpIgAMhOnMhhs8VsgtNlTvI B/WXkkNMFUwJjrCg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003rsa; t=1647978066; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JeCV7yssrEZvGRF+5cqew9vwgOGyRd2uJeYujMJjUyQ=; b=s74HO79NzBQrQwkW001LDUX4qu4x5IQV4jqQVkkb6EtaDar2LYUI0f8fRBaNCnXwvZcZCc 76cfP5AjyWPABAaXq2ysQQLTBrFL/3In+fHv2dOyFyp0jn9VcROUUWyIObY+SoubyPXu3w I1oURa1RgnADzs6vn/7jvOz0rwFNAhnpzNA3upptXJS3xotPdKqhSIVWIWM2CAWMJjKKQ8 In0H/aKADiUg2AFHaaXNF/A1EOwIoJYj9fKaRGhBTmq2LaWYcaZnO0dxfm6qxV2MJY4Twq oVZoKUUbZ5L1CH6zTBSDxBDiC4TD7LywFKiTc/EwfhZUuzL5pj+qhwN6A4eAcg== From: Stefan Schantl To: development@lists.ipfire.org Subject: [PATCH 2/5] ids-functions.pl: Allow "5" download attempts for each provider before fail. Date: Tue, 22 Mar 2022 20:40:56 +0100 Message-Id: <20220322194059.3030-2-stefan.schantl@ipfire.org> In-Reply-To: <20220322194059.3030-1-stefan.schantl@ipfire.org> References: <20220322194059.3030-1-stefan.schantl@ipfire.org> MIME-Version: 1.0 X-BeenThere: development@lists.ipfire.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: IPFire development talk List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: development-bounces@lists.ipfire.org Sender: "Development" Signed-off-by: Stefan Schantl --- config/cfgroot/ids-functions.pl | 38 ++++++++++++++++++++++++--------- 1 file changed, 28 insertions(+), 10 deletions(-) diff --git a/config/cfgroot/ids-functions.pl b/config/cfgroot/ids-functions.pl index eb276030b..c8bc52b1b 100644 --- a/config/cfgroot/ids-functions.pl +++ b/config/cfgroot/ids-functions.pl @@ -256,6 +256,10 @@ sub downloadruleset ($) { # If no provider is given default to "all". $provider //= 'all'; + # The amount of download attempts before giving up and + # logging an error. + my $max_dl_attempts = 5; + # Hash to store the providers and access id's, for which rules should be downloaded. my %sheduled_providers = (); @@ -364,19 +368,33 @@ sub downloadruleset ($) { # Pass the requested url to the downloader. my $request = HTTP::Request->new(GET => $url); - # Perform the request and save the output into the tmpfile. - my $response = $downloader->request($request, $tmpfile); + my $dl_attempt = 1; + my $response; - # Check if there was any error. - unless ($response->is_success) { - # Obtain error. - my $error = $response->content; + # Download and retry on failure. + while ($dl_attempt <= $max_dl_attempts) { + # Perform the request and save the output into the tmpfile. + $response = $downloader->request($request, $tmpfile); - # Log error message. - &_log_to_syslog("Unable to download the ruleset. \($error\)"); + # Check if the download was successfull. + if($response->is_success) { + # Break loop. + last; - # Return "1" - false. - return 1; + # Check if we ran out of download re-tries. + } elsif ($dl_attempt eq $max_dl_attempts) { + # Obtain error. + my $error = $response->content; + + # Log error message. + &_log_to_syslog("Unable to download the ruleset. \($error\)"); + + # Return "1" - false. + return 1; + } + + # Increase download attempt counter. + $dl_attempt++; } # Obtain the connection headers. From patchwork Tue Mar 22 19:40:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Schantl X-Patchwork-Id: 5387 Return-Path: Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by web04.haj.ipfire.org (Postfix) with ESMTPS id 4KNML22RDDz3xlr for ; Tue, 22 Mar 2022 19:41:14 +0000 (UTC) Received: from mail02.haj.ipfire.org (mail02.haj.ipfire.org [172.28.1.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail02.haj.ipfire.org", Issuer "R3" (verified OK)) by mail01.ipfire.org (Postfix) with ESMTPS id 4KNMKy1CdTz5Qs; Tue, 22 Mar 2022 19:41:10 +0000 (UTC) Received: from mail02.haj.ipfire.org (localhost [127.0.0.1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4KNMKx3clCz2yxL; Tue, 22 Mar 2022 19:41:09 +0000 (UTC) Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4KNMKv2Ndnz2xXd for ; Tue, 22 Mar 2022 19:41:07 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by mail01.ipfire.org (Postfix) with ESMTPSA id 4KNMKv0883z1Lp; Tue, 22 Mar 2022 19:41:06 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003ed25519; t=1647978067; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FNklpXOAyXg6LctDs9JNxASxGsaRrgKxAitoWULYZhY=; b=pZh9kOivKZxdnsm/iO8AsVs/ZnAHankiUv2ZfLM/CSzIxTsn3qeuDwXNop8JQfIQPCZvW3 fydOM/ONguVeagDw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003rsa; t=1647978067; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FNklpXOAyXg6LctDs9JNxASxGsaRrgKxAitoWULYZhY=; b=M1sMwtA8lJ1posEIecMb4CYy8d+yiyZFsoHl1zKup6gm+DtzizMylIwaLI4oOFj90PO39u qn237vq8mLF+HW5gpZiwrwf5iOtcqDnIc7VfUtuBvDmkr1vd7RcYOHRkAKqRR3iZoDX7Pn R9rByFqCA2GKS0VbcMtzNfLS6ggiTh+ngZVQGVQTWP6LNZ3/xpzLiWezhS2dwBdMFvh8CU K34OohTW7XUm5pwBPKL/dNuflvTwGpb0CRAtHRVkzar6Q0pK0RuMdjW7P9sos3vL9D6KVI nIv8enN1T5gF4VExJrLUc49cJpLeSCDE7MQ7MX0EtaazOsELgsbbAKCFLPpdBg== From: Stefan Schantl To: development@lists.ipfire.org Subject: [PATCH 3/5] ids-functions.pl: Remove temporary file, if the download failed. Date: Tue, 22 Mar 2022 20:40:57 +0100 Message-Id: <20220322194059.3030-3-stefan.schantl@ipfire.org> In-Reply-To: <20220322194059.3030-1-stefan.schantl@ipfire.org> References: <20220322194059.3030-1-stefan.schantl@ipfire.org> MIME-Version: 1.0 X-BeenThere: development@lists.ipfire.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: IPFire development talk List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: development-bounces@lists.ipfire.org Sender: "Development" Signed-off-by: Stefan Schantl --- config/cfgroot/ids-functions.pl | 3 +++ 1 file changed, 3 insertions(+) diff --git a/config/cfgroot/ids-functions.pl b/config/cfgroot/ids-functions.pl index c8bc52b1b..dfbeb1a7d 100644 --- a/config/cfgroot/ids-functions.pl +++ b/config/cfgroot/ids-functions.pl @@ -393,6 +393,9 @@ sub downloadruleset ($) { return 1; } + # Remove temporary file, if one exists. + unlink("$tmpfile") if (-e "$tmpfile"); + # Increase download attempt counter. $dl_attempt++; } From patchwork Tue Mar 22 19:40:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Schantl X-Patchwork-Id: 5386 Return-Path: Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by web04.haj.ipfire.org (Postfix) with ESMTPS id 4KNML15HwBz3xqd for ; Tue, 22 Mar 2022 19:41:13 +0000 (UTC) Received: from mail02.haj.ipfire.org (mail02.haj.ipfire.org [172.28.1.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail02.haj.ipfire.org", Issuer "R3" (verified OK)) by mail01.ipfire.org (Postfix) with ESMTPS id 4KNMKy4DMtz5V3; Tue, 22 Mar 2022 19:41:10 +0000 (UTC) Received: from mail02.haj.ipfire.org (localhost [127.0.0.1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4KNMKx4Bf4z2yxs; Tue, 22 Mar 2022 19:41:09 +0000 (UTC) Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4KNMKv5K91z2xWt for ; Tue, 22 Mar 2022 19:41:07 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by mail01.ipfire.org (Postfix) with ESMTPSA id 4KNMKv2rpyzLX; Tue, 22 Mar 2022 19:41:07 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003ed25519; t=1647978067; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=uMxCrqyDrzrfCF/FVN1UL4sgIdLUK1vkwKaR7OmvQ4k=; b=CWgC1ZqFL87lF3NspdSt1OmyNbYLhKFDCKTh2WUF6lKtDKoyDEUzfx58TxJKLM9T9crUyX 4Vq3M2mHp+hyTzBw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003rsa; t=1647978067; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=uMxCrqyDrzrfCF/FVN1UL4sgIdLUK1vkwKaR7OmvQ4k=; b=fzdx27JIPhndXb+LmWYM+vcu1HC/BbRpb5O7x5Y3TEPiyPTb+GR2AIyhxyWQlC2sL1J4g6 BpPoEeouZgjR0GsftJGLiiojFDsIofaqwh+xe5NKCj3bOYeRW+tu+BM7lBaQzJCwHHgovb sdZKKdP/42Hwa0RoEBNLphjbW62EGtoNdJy4RRfDy0wtWA6p+O8BdMGZqQAClmdfFig3gT ui/9Ve4r5oOzju5ZCW7Km4VHOFih/8Gsd3nlBdMAl8C9uDkgQlQgIvjO6hS8D9bZbSKbrP +LwEhuAmUuUTlERg7y7HKx/us1l4JLc0SKVhzZc463qnBVxs/CNPQ5bNsp17VQ== From: Stefan Schantl To: development@lists.ipfire.org Subject: [PATCH 4/5] ids-functions.pl: Use If-Modified-Since header to reduce file downloads. Date: Tue, 22 Mar 2022 20:40:58 +0100 Message-Id: <20220322194059.3030-4-stefan.schantl@ipfire.org> In-Reply-To: <20220322194059.3030-1-stefan.schantl@ipfire.org> References: <20220322194059.3030-1-stefan.schantl@ipfire.org> MIME-Version: 1.0 X-BeenThere: development@lists.ipfire.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: IPFire development talk List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: development-bounces@lists.ipfire.org Sender: "Development" When using the "If-Modified-Since" header, the server can be requested if a modified version of the file can be served. In case that is true, the file will be sent and stored by the downloader function. If the file has not been touched since the last time, the server will respond with the code "304" (Not modified). This tells us, that the current stored file is the latest one (still up-to-date) and we safely can skip the download attempt for this provider. Signed-off-by: Stefan Schantl --- config/cfgroot/ids-functions.pl | 38 ++++++++++++++++++++++++++++++--- 1 file changed, 35 insertions(+), 3 deletions(-) diff --git a/config/cfgroot/ids-functions.pl b/config/cfgroot/ids-functions.pl index dfbeb1a7d..d7df41dd1 100644 --- a/config/cfgroot/ids-functions.pl +++ b/config/cfgroot/ids-functions.pl @@ -365,9 +365,25 @@ sub downloadruleset ($) { my $tmp = File::Temp->new( SUFFIX => ".tmp", DIR => "/var/tmp/", UNLINK => 0 ); my $tmpfile = $tmp->filename(); + # Genarate and assign file name and path to store the downloaded rules file. + my $dl_rulesfile = &_get_dl_rulesfile($provider); + + # Load perl module to deal with file atributes. + use File::stat; + + # Get the mtime of the rulesfile if it exists. + my $mtime = (stat($dl_rulesfile)->mtime) if (-f $dl_rulesfile); + + # Convert the mtime into gmtime format. + my $gmtime = gmtime($mtime || 0); + # Pass the requested url to the downloader. my $request = HTTP::Request->new(GET => $url); + # Add the If-Modified-Since header to the request, containing the omited and converted + # mtime of the downloaded rules file, if one is present. + $request->header( 'If-Modified-Since' => "$gmtime" ); + my $dl_attempt = 1; my $response; @@ -381,6 +397,14 @@ sub downloadruleset ($) { # Break loop. last; + # Check if the server responds with 304 (Not Modified). + } elsif ($response->code == 304) { + # Log to syslog. + &_log_to_syslog("Ruleset is up-to-date, no update required."); + + # Nothing to be done, the ruleset is up-to-date. + return; + # Check if we ran out of download re-tries. } elsif ($dl_attempt eq $max_dl_attempts) { # Obtain error. @@ -406,6 +430,10 @@ sub downloadruleset ($) { # Get the remote size of the downloaded file. my $remote_filesize = $headers->content_length; + # Get the timestamp from header, when the file has been modified the + # last time. + my $last_modified = $headers->last_modified; + # Load perl stat module. use File::stat; @@ -428,9 +456,6 @@ sub downloadruleset ($) { return 1; } - # Genarate and assign file name and path to store the downloaded rules file. - my $dl_rulesfile = &_get_dl_rulesfile($provider); - # Check if a file name could be obtained. unless ($dl_rulesfile) { # Log error message. @@ -449,6 +474,13 @@ sub downloadruleset ($) { # Overwrite the may existing rulefile or tarball with the downloaded one. move("$tmpfile", "$dl_rulesfile"); + # Check if the server respond contained a last_modified value. + if ($last_modified) { + # Assign the last modified timestamp from server as mtime to the + # rules file. + utime(time(), "$last_modified", "$dl_rulesfile"); + } + # Delete temporary file. unlink("$tmpfile"); From patchwork Tue Mar 22 19:40:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Schantl X-Patchwork-Id: 5388 Return-Path: Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by web04.haj.ipfire.org (Postfix) with ESMTPS id 4KNML66Mstz3xlr for ; Tue, 22 Mar 2022 19:41:18 +0000 (UTC) Received: from mail02.haj.ipfire.org (mail02.haj.ipfire.org [172.28.1.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail02.haj.ipfire.org", Issuer "R3" (verified OK)) by mail01.ipfire.org (Postfix) with ESMTPS id 4KNMKy71RYz5Vj; Tue, 22 Mar 2022 19:41:10 +0000 (UTC) Received: from mail02.haj.ipfire.org (localhost [127.0.0.1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4KNMKx4n0Xz2yyV; Tue, 22 Mar 2022 19:41:09 +0000 (UTC) Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4KNMKw0qXqz2xWt for ; Tue, 22 Mar 2022 19:41:08 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by mail01.ipfire.org (Postfix) with ESMTPSA id 4KNMKv5y7Bz1Lp; Tue, 22 Mar 2022 19:41:07 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003ed25519; t=1647978068; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kY63ojOVGRfh+iW7CfHQt9NMGzG6f3GZStpDxEaisgg=; b=XtiEfxEMpr9hoYPXA8fMyxpFuHzk5M3pKLvkPfZNPJ+zwAXTBiSbNwxu/jtgHrcpbOnnPk ZKmIoPDczg7vtrCA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003rsa; t=1647978068; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kY63ojOVGRfh+iW7CfHQt9NMGzG6f3GZStpDxEaisgg=; b=q/TZ/cepYzEk5laP96YiDaZZnwlMeFYZcjpYow520/94OXhZx/xKoaGfzIi9ejXcwHEFw7 kmj3rKznMRj4VT5SRTPsZ3Ck7NeTwxCOlRXirSMFIn1Oz3bxYxCKL4oIQ8WtXu1EoEAfai PF6iL8OWt9RyiskapOfbZEWLClpwpMllwDmlDv5xpPS9B0TZHdCWBgijuUmQIjJKkwlK+O HWHlmg/8bKzJLro3Agsjrj3E5MqcElj3f0WFV5lj1VFmWbAg+dNAw2Ncqc4/X5SGNRdo7a gauwiquAx9UVPcUkmomchOr7Bf/c8ceqwK7aSVtQ36u+07o2SCBG+/iKAxnU9w== From: Stefan Schantl To: development@lists.ipfire.org Subject: [PATCH 5/5] ids-functions.pl: Do not longer call any log message as "ERROR". Date: Tue, 22 Mar 2022 20:40:59 +0100 Message-Id: <20220322194059.3030-5-stefan.schantl@ipfire.org> In-Reply-To: <20220322194059.3030-1-stefan.schantl@ipfire.org> References: <20220322194059.3030-1-stefan.schantl@ipfire.org> MIME-Version: 1.0 X-BeenThere: development@lists.ipfire.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: IPFire development talk List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: development-bounces@lists.ipfire.org Sender: "Development" Signed-off-by: Stefan Schantl --- config/cfgroot/ids-functions.pl | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/config/cfgroot/ids-functions.pl b/config/cfgroot/ids-functions.pl index d7df41dd1..9eb375bc9 100644 --- a/config/cfgroot/ids-functions.pl +++ b/config/cfgroot/ids-functions.pl @@ -226,7 +226,7 @@ sub checkdiskspace () { # Check if the available disk space is more than 300MB. if ($available < 300) { # Log error to syslog. - &_log_to_syslog("Not enough free disk space on /var. Only $available MB from 300 MB available."); + &_log_to_syslog(" Not enough free disk space on /var. Only $available MB from 300 MB available."); # Exit function and return "1" - False. return 1; @@ -270,7 +270,7 @@ sub downloadruleset ($) { # Check if a ruleset has been configured. unless(%used_providers) { # Log that no ruleset has been configured and abort. - &_log_to_syslog("No ruleset provider has been configured."); + &_log_to_syslog(" No ruleset provider has been configured."); # Return "1". return 1; @@ -333,7 +333,7 @@ sub downloadruleset ($) { # Loop through the hash of sheduled providers. foreach my $provider ( keys %sheduled_providers) { # Log download/update of the ruleset. - &_log_to_syslog("Downloading ruleset for provider: $provider."); + &_log_to_syslog(" Downloading ruleset for provider: $provider."); # Grab the download url for the provider. my $url = $IDS::Ruleset::Providers{$provider}{'dl_url'}; @@ -354,7 +354,7 @@ sub downloadruleset ($) { # Abort if no url could be determined for the provider. unless ($url) { # Log error and abort. - &_log_to_syslog("Unable to gather a download URL for the selected ruleset provider."); + &_log_to_syslog(" Unable to gather a download URL for the selected ruleset provider."); return 1; } @@ -400,7 +400,7 @@ sub downloadruleset ($) { # Check if the server responds with 304 (Not Modified). } elsif ($response->code == 304) { # Log to syslog. - &_log_to_syslog("Ruleset is up-to-date, no update required."); + &_log_to_syslog(" Ruleset is up-to-date, no update required."); # Nothing to be done, the ruleset is up-to-date. return; @@ -411,7 +411,7 @@ sub downloadruleset ($) { my $error = $response->content; # Log error message. - &_log_to_syslog("Unable to download the ruleset. \($error\)"); + &_log_to_syslog(" Unable to download the ruleset. \($error\)"); # Return "1" - false. return 1; @@ -446,8 +446,8 @@ sub downloadruleset ($) { # Check if both file sizes match. if (($remote_filesize) && ($remote_filesize ne $local_filesize)) { # Log error message. - &_log_to_syslog("Unable to completely download the ruleset. "); - &_log_to_syslog("Only got $local_filesize Bytes instead of $remote_filesize Bytes. "); + &_log_to_syslog(" Unable to completely download the ruleset. "); + &_log_to_syslog(" Only got $local_filesize Bytes instead of $remote_filesize Bytes. "); # Delete temporary file. unlink("$tmpfile"); @@ -459,7 +459,7 @@ sub downloadruleset ($) { # Check if a file name could be obtained. unless ($dl_rulesfile) { # Log error message. - &_log_to_syslog("Unable to store the downloaded rules file. "); + &_log_to_syslog(" Unable to store the downloaded rules file. "); # Delete downloaded temporary file. unlink("$tmpfile"); @@ -518,7 +518,7 @@ sub extractruleset ($) { # Check if the file exists. unless (-f $tarball) { - &_log_to_syslog("Could not find ruleset file: $tarball"); + &_log_to_syslog(" Could not find ruleset file: $tarball"); # Return nothing. return; @@ -897,7 +897,7 @@ sub _log_to_syslog ($) { # The syslog function works best with an array based input, # so generate one before passing the message details to syslog. - my @syslog = ("ERR", " $message"); + my @syslog = ("ERR", "$message"); # Establish the connection to the syslog service. openlog('oinkmaster', 'cons,pid', 'user');