From patchwork Fri Apr 18 10:54:40 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Schantl X-Patchwork-Id: 8612 Return-Path: Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) client-signature RSA-PSS (4096 bits)) (Client CN "mail01.haj.ipfire.org", Issuer "R10" (verified OK)) by web04.haj.ipfire.org (Postfix) with ESMTPS id 4ZfC165zgKz3x1X for ; Fri, 18 Apr 2025 11:18:02 +0000 (UTC) Received: from mail02.haj.ipfire.org (mail02.haj.ipfire.org [172.28.1.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) client-signature ECDSA (secp384r1)) (Client CN "mail02.haj.ipfire.org", Issuer "E5" (verified OK)) by mail01.ipfire.org (Postfix) with ESMTPS id 4ZfC146Ry8z34J for ; Fri, 18 Apr 2025 11:18:00 +0000 (UTC) Received: from mail02.haj.ipfire.org (localhost [127.0.0.1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4ZfC145kY5z337h for ; Fri, 18 Apr 2025 11:18:00 +0000 (UTC) X-Original-To: development@lists.ipfire.org Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) client-signature RSA-PSS (4096 bits)) (Client CN "mail01.haj.ipfire.org", Issuer "R10" (verified OK)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4ZfC115Nv4z3359 for ; Fri, 18 Apr 2025 11:17:57 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail01.ipfire.org (Postfix) with ESMTPSA id 4ZfC111jfKz10s; Fri, 18 Apr 2025 11:17:57 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003ed25519; t=1744975077; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jKDnCt9YPAbEzCzPCvS/zBAmZ23oRAjUTRtcVWYnhfA=; b=GaX1pupMvw3YMABUJDp2fyYAN6uUVkg2Cj2vP/rgy1ScUl04y4wZjjIbLL1oWK5PCCgZwL nQ7J/RnfOS75odAg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003rsa; t=1744975077; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jKDnCt9YPAbEzCzPCvS/zBAmZ23oRAjUTRtcVWYnhfA=; b=KhGeDaomBrDKIc1agsCOq7wHd2TvG15bQtaQnbnsOrK5lRnQCr6msQ4vR9o9Bad33HYJgV ubgjeUw4aU2AUYxgdYlNnKy7KbDLRHbHxX1gDLz9SAhq7cqlerzV2uvuVKeo0u+wwaaNsa 1bIXvrSkuhq7UFv3CGLUIJtnj+aXXXJkYjhZK69/2gPYsx9WwZo/NVYrXQOlgzd4LiXVMf 6KnMs3a+egoyuKUHkAp0DZ/N/RwY8KhtFBMaVdEro7xH+u6At1NzcupXPZ6B3Y64ZZYTV+ kNWR39U6PkDc2Q3Ze/NAMHPPfXNSYo5K/Uy0SQHqrpk407/37H3KLnW3XJMdaA== From: Stefan Schantl To: development@lists.ipfire.org Cc: Stefan Schantl Subject: [PATCHv2 1/7] http-client-functions.pl: Introduce LWP-based flexible downloader function. Date: Fri, 18 Apr 2025 12:54:40 +0200 Message-ID: <20250418110741.7756-2-stefan.schantl@ipfire.org> In-Reply-To: <20250418110741.7756-1-stefan.schantl@ipfire.org> References: <20250418110741.7756-1-stefan.schantl@ipfire.org> Precedence: list List-Id: List-Subscribe: , List-Unsubscribe: , List-Post: List-Help: Sender: Mail-Followup-To: MIME-Version: 1.0 This perl library contains a function which can be used to grab content and/or store it into files. Signed-off-by: Stefan Schantl --- config/cfgroot/http-client-functions.pl | 290 ++++++++++++++++++++++++ config/rootfiles/common/configroot | 1 + lfs/configroot | 1 + 3 files changed, 292 insertions(+) create mode 100644 config/cfgroot/http-client-functions.pl diff --git a/config/cfgroot/http-client-functions.pl b/config/cfgroot/http-client-functions.pl new file mode 100644 index 000000000..26ead6908 --- /dev/null +++ b/config/cfgroot/http-client-functions.pl @@ -0,0 +1,290 @@ +#!/usr/bin/perl -w +############################################################################ +# # +# This file is part of the IPFire Firewall. # +# # +# IPFire is free software; you can redistribute it and/or modify # +# it under the terms of the GNU General Public License as published by # +# the Free Software Foundation; either version 2 of the License, or # +# (at your option) any later version. # +# # +# IPFire is distributed in the hope that it will be useful, # +# but WITHOUT ANY WARRANTY; without even the implied warranty of # +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # +# GNU General Public License for more details. # +# # +# You should have received a copy of the GNU General Public License # +# along with IPFire; if not, write to the Free Software # +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # +# # +# Copyright (C) 2025 IPFire Team # +# # +############################################################################ + +package HTTPClient; + +require '/var/ipfire/general-functions.pl'; + +use strict; + +# Load module to move files. +use File::Copy; + +# Load module to get file stats. +use File::stat; + +# Load module to deal with temporary files. +use File::Temp; + +# Load module to deal with the date formats used by the HTTP protocol. +use HTTP::Date; + +# Load the libwwwperl User Agent module. +use LWP::UserAgent; + +# Function to grab a given URL content or to download and store it on disk. +# +# The function requires a configuration hash to be passed. +# +# The following options (hash keys) are supported: +# +# URL -> The URL to the content or file. REQUIRED! +# FILE -> The filename as fullpath where the content/file should be stored on disk. +# ETAGSFILE -> A filename again as fullpath where Etags should be stored and read. +# ETAGPREFIX -> In case a custom etag name should be used, otherwise it defaults to the given URL. +# MAXSIZE -> In bytes until the downloader will abort downloading. (example: 10_485_760 for 10MB) +# +# If a file is given an If-Modified-Since header will be generated from the last modified timestamp +# of an already stored file. In case an Etag file is specified an If-None-Match header will be added to +# the request - Both can be used at the same time. +# +# In case no FILE option has been passed to the function, the content of the requested URL will be returned. +# +# Return codes (if FILE is used): +# +# nothing - On success +# no url - If no URL has been specified. +# not_modified - In case the servers responds with "Not modified" (304) +# dl_error - If the requested URL cannot be accessed. +# incomplete download - In case the size of the local file does not match the remote content_lenght. +# +sub downloader (%) { + my (%args) = @_; + + # Remap args hash and convert all keys into upper case format. + %args = map { uc $_ => $args{$_} } keys %args; + + # The amount of download attempts before giving up and + # logging an error. + my $max_dl_attempts = 3; + + # Temporary directory to download the files. + my $tmp_dl_directory = "/var/tmp"; + + # Assign hash values. + my $url = $args{"URL"} if (exists($args{"URL"})); + my $file = $args{"FILE"} if (exists($args{"FILE"})); + my $etags_file = $args{"ETAGSFILE"} if (exists($args{"ETAGSFILE"})); + my $etagprefix = $url; + $etagprefix = $args{"ETAGPREFIX"} if (exists($args{"ETAGPREFIX"})); + my $max_size = $args{"MAXSIZE"} if (exists($args{"MAXSIZE"})); + + # Abort with error "no url", if no URL has been given. + die "downloader: No URL has been given." unless ($url); + + my %etags = (); + my $tmpfile; + + # Read-in proxysettings. + my %proxysettings=(); + &General::readhash("${General::swroot}/proxy/settings", \%proxysettings); + + # Create a user agent instance. + # + # Request SSL hostname verification and specify path + # to the CA file. + my $ua = LWP::UserAgent->new( + ssl_opts => { + SSL_ca_file => '/etc/ssl/cert.pem', + verify_hostname => 1, + }, + ); + + # Set timeout to 10 seconds. + $ua->timeout(10); + + # Assign maximum download size if set. + $ua->max_size($max_size) if ($max_size); + + # Generate UserAgent. + my $agent = &General::MakeUserAgent(); + + # Set the generated UserAgent. + $ua->agent($agent); + + # Check if an upstream proxy is configured. + if ($proxysettings{'UPSTREAM_PROXY'}) { + # Start generating proxy url. + my $proxy_url = "http://"; + + # Check if the proxy requires authentication. + if (($proxysettings{'UPSTREAM_USER'}) && ($proxysettings{'UPSTREAM_PASSWORD'})) { + # Add proxy auth details. + $proxy_url .= "$proxysettings{'UPSTREAM_USER'}\:$proxysettings{'UPSTREAM_PASSWORD'}\@"; + } + + # Add proxy server address and port. + $proxy_url .= $proxysettings{'UPSTREAM_PROXY'}; + + # Append proxy settings. + $ua->proxy(['http', 'https'], $proxy_url); + } + + # Create a HTTP request element and pass the given URL to it. + my $request = HTTP::Request->new(GET => $url); + + # Check if a file to store the output has been provided. + if ($file) { + # Check if the given file already exits, because it has been downloaded in the past. + # + # In this case we are requesting the server if the remote file has been changed or not. + # This will be done by sending the modification time in a special HTTP header. + if (-f $file) { + # Call stat on the file. + my $stat = stat($file); + + # Omit the mtime of the existing file. + my $mtime = $stat->mtime; + + # Convert the timestamp into right format. + my $http_date = time2str($mtime); + + # Add the If-Modified-Since header to the request to ask the server if the + # file has been modified. + $request->header( 'If-Modified-Since' => "$http_date" ); + } + + # Generate a temporary file name, located in the tempoary download directory and with a suffix of ".tmp". + # The downloaded file will be stored there until some sanity checks are performed. + my $tmp = File::Temp->new( SUFFIX => ".tmp", DIR => "$tmp_dl_directory/", UNLINK => 0 ); + $tmpfile = $tmp->filename(); + } + + # Check if an file for etags has been given. + if ($etags_file) { + # Read-in Etags file for known Etags if the file is present. + &readhash("$etags_file", \%etags) if (-f $etags_file); + + # Check if an Etag for the requested file is stored. + if ($etags{$etagprefix}) { + # Grab the stored tag. + my $etag = $etags{$etagprefix}; + + # Add an "If-None-Match header to the request to ask the server if the + # file has been modified. + $request->header( 'If-None-Match' => $etag ); + } + } + + my $dl_attempt = 1; + my $response; + + # Download and retry on failure. + while ($dl_attempt <= $max_dl_attempts) { + # Perform the request and save the output into the tmpfile if requested. + $response = $ua->request($request, $tmpfile); + + # Check if the download was successfull. + if($response->is_success) { + # Break loop. + last; + + # Check if the server responds with 304 (Not Modified). + } elsif ($response->code == 304) { + # Remove temporary file, if one exists. + unlink("$tmpfile") if (-e "$tmpfile"); + + # Return "not modified". + return "not modified"; + + # Check if we ran out of download re-tries. + } elsif ($dl_attempt eq $max_dl_attempts) { + # Obtain error. + my $error = $response->content; + + # Remove temporary file, if one exists. + unlink("$tmpfile") if (-e "$tmpfile"); + + # Return the error message from response.. + return "$error"; + } + + # Remove temporary file, if one exists. + unlink("$tmpfile") if (-e "$tmpfile"); + + # Increase download attempt counter. + $dl_attempt++; + } + + # Obtain the connection headers. + my $headers = $response->headers; + + # Check if an Etag file has been provided. + if ($etags_file) { + # Grab the Etag from the response if the server provides one. + if ($response->header('Etag')) { + # Add the provided Etag to the hash of tags. + $etags{$etagprefix} = $response->header('Etag'); + + # Write the etags file. + &General::writehash($etags_file, \%etags); + } + } + + # Check if the response should be stored on disk. + if ($file) { + # Get the remote size of the content. + my $remote_size = $response->header('Content-Length'); + + # Perform a stat on the temporary file. + my $stat = stat($tmpfile); + + # Grab the size of the stored temporary file. + my $local_size = $stat->size; + + # Check if both sizes are equal. + if(($remote_size) && ($remote_size ne $local_size)) { + # Delete the temporary file. + unlink("$tmpfile"); + + # Abort and return "incomplete download" as error. + return "incomplete download"; + } + + # Move the temporaray file to the desired file by overwriting a may + # existing one. + move("$tmpfile", "$file"); + + # Omit the timestamp from response header, when the file has been modified the + # last time. + my $last_modified = $headers->last_modified; + + # Check if we got a last-modified value from the server. + if ($last_modified) { + # Assign the last-modified timestamp as mtime to the + # stored file. + utime(time(), "$last_modified", "$file"); + } + + # Delete temporary file. + unlink("$tmpfile"); + + # If we got here, everything worked fine. Return nothing. + return; + } else { + # Decode the response content and return it. + return $response->decoded_content; + } +} + +1; diff --git a/config/rootfiles/common/configroot b/config/rootfiles/common/configroot index 9839eee45..51472e7c5 100644 --- a/config/rootfiles/common/configroot +++ b/config/rootfiles/common/configroot @@ -79,6 +79,7 @@ var/ipfire/fwlogs var/ipfire/general-functions.pl var/ipfire/graphs.pl var/ipfire/header.pl +var/ipfire/http-client-functions.pl var/ipfire/location-functions.pl var/ipfire/ids-functions.pl var/ipfire/ipblocklist-functions.pl diff --git a/lfs/configroot b/lfs/configroot index 9f6c1ff8c..1f752ddb6 100644 --- a/lfs/configroot +++ b/lfs/configroot @@ -76,6 +76,7 @@ $(TARGET) : # Copy initial configfiles cp $(DIR_SRC)/config/cfgroot/header.pl $(CONFIG_ROOT)/ + cp $(DIR_SRC)/config/cfgroot/http-client-functions.pl $(CONFIG_ROOT)/ cp $(DIR_SRC)/config/cfgroot/general-functions.pl $(CONFIG_ROOT)/ cp $(DIR_SRC)/config/cfgroot/network-functions.pl $(CONFIG_ROOT)/ cp $(DIR_SRC)/config/cfgroot/location-functions.pl $(CONFIG_ROOT)/ From patchwork Fri Apr 18 10:54:41 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Schantl X-Patchwork-Id: 8613 Return-Path: Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) client-signature RSA-PSS (4096 bits)) (Client CN "mail01.haj.ipfire.org", Issuer "R10" (verified OK)) by web04.haj.ipfire.org (Postfix) with ESMTPS id 4ZfC182SDMz3wb5 for ; Fri, 18 Apr 2025 11:18:04 +0000 (UTC) Received: from mail02.haj.ipfire.org (mail02.haj.ipfire.org [172.28.1.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) client-signature ECDSA (secp384r1)) (Client CN "mail02.haj.ipfire.org", Issuer "E5" (verified OK)) by mail01.ipfire.org (Postfix) with ESMTPS id 4ZfC161Mpcz3N7 for ; Fri, 18 Apr 2025 11:18:02 +0000 (UTC) Received: from mail02.haj.ipfire.org (localhost [127.0.0.1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4ZfC1560vdz337b for ; Fri, 18 Apr 2025 11:18:01 +0000 (UTC) X-Original-To: development@lists.ipfire.org Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) client-signature RSA-PSS (4096 bits)) (Client CN "mail01.haj.ipfire.org", Issuer "R10" (verified OK)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4ZfC126C7Jz336p for ; Fri, 18 Apr 2025 11:17:58 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail01.ipfire.org (Postfix) with ESMTPSA id 4ZfC1211ylz2yn; Fri, 18 Apr 2025 11:17:58 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003ed25519; t=1744975078; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KIaaJAJxnhPQczA6FFCL5OC7kP89UPy3POzTNjCaKns=; b=Eu9tXWMpMa2PBkEMyxFBFJg8Poy1O81q4w5ewy3OU5g5aVYW9dHkFsh+qYJoKeknUPj4i3 o+IIY6bscFw6gYAQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003rsa; t=1744975078; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KIaaJAJxnhPQczA6FFCL5OC7kP89UPy3POzTNjCaKns=; b=ns0pVkX37JGIA162LdLtiRIS5zTW/JEAUHuigGq4SuwEIU3SkAByiQGiT1HH6AtXsJz1uH JHV7EcwRimeo4G8r46rbC5piFx2FyRY5CzLROiOG276LwIJzU1oIvHgnYq2Hs+aUPrx39Y Gh9P/xyaL/AQVN6QhS6KXMRnHSEKYEXY8JwNT3EI515RMNbeuEgU+KBv/fzCiDuJAMpj14 O/g6ET/i0tBiHub+3x8TIulRKgp6aq+o/xPmHI+EFZnylKThJOi2QRCorQeASyHz+SoMBw UHG9ndWrI81y3DkaIFpKLj23nWE6oHK+MaRgg10JRzYw/iDulB1uDhbidQn7xQ== From: Stefan Schantl To: development@lists.ipfire.org Cc: Stefan Schantl Subject: [PATCHv2 2/7] http-client-functions.pl: Add FetchPublicIP function. Date: Fri, 18 Apr 2025 12:54:41 +0200 Message-ID: <20250418110741.7756-3-stefan.schantl@ipfire.org> In-Reply-To: <20250418110741.7756-1-stefan.schantl@ipfire.org> References: <20250418110741.7756-1-stefan.schantl@ipfire.org> Precedence: list List-Id: List-Subscribe: , List-Unsubscribe: , List-Post: List-Help: Sender: Mail-Followup-To: MIME-Version: 1.0 This functions uses the newly introduced downloader to fetch the pulic IPv4 address on red and will replace the current used one from the general-functions.pl library. Signed-off-by: Stefan Schantl --- config/cfgroot/http-client-functions.pl | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/config/cfgroot/http-client-functions.pl b/config/cfgroot/http-client-functions.pl index 26ead6908..bfb9fdd20 100644 --- a/config/cfgroot/http-client-functions.pl +++ b/config/cfgroot/http-client-functions.pl @@ -287,4 +287,24 @@ sub downloader (%) { } } +# +# Tiny function to grab the public red IPv4 address using LWL. +# +sub FetchPublicIp { + # URL to grab the public IP. + my $url = "https://checkip4.dns.lightningwirelabs.com"; + + # Call downloader to fetch the public IP. + my $response = &downloader("URL" => $url); + + # Omit the address from the resonse message. + if ($response =~ /Your IP address is: (\d+.\d+.\d+.\d+)/) { + # Return the address. + return $1; + } + + # Unable to grab the address - Return nothing. + return; +} + 1; From patchwork Fri Apr 18 10:54:42 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Schantl X-Patchwork-Id: 8614 Return-Path: Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) client-signature RSA-PSS (4096 bits)) (Client CN "mail01.haj.ipfire.org", Issuer "R10" (verified OK)) by web04.haj.ipfire.org (Postfix) with ESMTPS id 4ZfC1954Y0z3wb5 for ; Fri, 18 Apr 2025 11:18:05 +0000 (UTC) Received: from mail02.haj.ipfire.org (mail02.haj.ipfire.org [172.28.1.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) client-signature ECDSA (secp384r1)) (Client CN "mail02.haj.ipfire.org", Issuer "E5" (verified OK)) by mail01.ipfire.org (Postfix) with ESMTPS id 4ZfC174VFnz3BW for ; Fri, 18 Apr 2025 11:18:03 +0000 (UTC) Received: from mail02.haj.ipfire.org (localhost [127.0.0.1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4ZfC166L9bz337n for ; Fri, 18 Apr 2025 11:18:02 +0000 (UTC) X-Original-To: development@lists.ipfire.org Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) client-signature RSA-PSS (4096 bits)) (Client CN "mail01.haj.ipfire.org", Issuer "R10" (verified OK)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4ZfC136fgZz335f for ; Fri, 18 Apr 2025 11:17:59 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail01.ipfire.org (Postfix) with ESMTPSA id 4ZfC123dRHz33G; Fri, 18 Apr 2025 11:17:58 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003ed25519; t=1744975078; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MGgtnekjB75HK2GkOdhaXzyVFNJAAsfXowkRCAV7opU=; b=XmKXM00d40LtCwem9mm5fTYv5ZJXLe7ywKsufz5qtboHfga4reBXtWvS7aD2U5kn5I4fH4 +51Xh6dDGVjFiYBg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003rsa; t=1744975078; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MGgtnekjB75HK2GkOdhaXzyVFNJAAsfXowkRCAV7opU=; b=wimMNALxZmSJVg69M0SqjdXD25tHA955c65ngk3SwZc2Kk6LFitx8F9+8rw6b0YYZchxfv uAfSJ6iMT4po/yevAFQjszqbOoRRcMCMvFgYtipMwZmdfK81a4yyrPeV55FOZIJYZG7JE8 LTn5zaEkVAqGBlWoj0rs6dkq+fXgXmq8o12BxSPZNNj+LLnbArGrXyf7VHGoP8vtU3PzCu HlzZ5d91iWr3ojFo5KqaB1B/M6c8wUC/QAIlLs7+iRD1cYu3KdIlsD/LDPdNP/KDXKw2P7 GXDYphNqz67kocTaj4s4Puo1z7TKEjp57wmTMHww7Schu98y6vsb6PlZHJw9FQ== From: Stefan Schantl To: development@lists.ipfire.org Cc: Stefan Schantl Subject: [PATCHv2 3/7] general-functions.pl: Drop FetchPublicIp function. Date: Fri, 18 Apr 2025 12:54:42 +0200 Message-ID: <20250418110741.7756-4-stefan.schantl@ipfire.org> In-Reply-To: <20250418110741.7756-1-stefan.schantl@ipfire.org> References: <20250418110741.7756-1-stefan.schantl@ipfire.org> Precedence: list List-Id: List-Subscribe: , List-Unsubscribe: , List-Post: List-Help: Sender: Mail-Followup-To: MIME-Version: 1.0 This function has been reworked and moved into the http-client-functions library. Signed-off-by: Stefan Schantl --- config/cfgroot/general-functions.pl | 21 --------------------- 1 file changed, 21 deletions(-) diff --git a/config/cfgroot/general-functions.pl b/config/cfgroot/general-functions.pl index 8ba6e3f79..861f95dec 100644 --- a/config/cfgroot/general-functions.pl +++ b/config/cfgroot/general-functions.pl @@ -17,7 +17,6 @@ package General; use strict; use Socket; use IO::Socket; -use Net::SSLeay; use Net::IPv4Addr qw(:all); $General::version = 'VERSION'; @@ -961,26 +960,6 @@ sub findhasharraykey { } } -sub FetchPublicIp { - my %proxysettings; - &General::readhash("${General::swroot}/proxy/settings", \%proxysettings); - if ($_=$proxysettings{'UPSTREAM_PROXY'}) { - my ($peer, $peerport) = (/^(?:[a-zA-Z ]+\:\/\/)?(?:[A-Za-z0-9\_\.\-]*?(?:\:[A-Za-z0-9\_\.\-]*?)?\@)?([a-zA-Z0-9\.\_\-]*?)(?:\:([0-9]{1,5}))?(?:\/.*?)?$/); - Net::SSLeay::set_proxy($peer,$peerport,$proxysettings{'UPSTREAM_USER'},$proxysettings{'UPSTREAM_PASSWORD'} ); - } - my $user_agent = &MakeUserAgent(); - my ($out, $response) = Net::SSLeay::get_http( 'checkip4.dns.lightningwirelabs.com', - 80, - "/", - Net::SSLeay::make_headers('User-Agent' => $user_agent ) - ); - if ($response =~ m%HTTP/1\.. 200 OK%) { - $out =~ /Your IP address is: (\d+.\d+.\d+.\d+)/; - return $1; - } - return ''; -} - # # Check if hostname.domain provided have IP provided # use gethostbyname to verify that From patchwork Fri Apr 18 10:54:43 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Schantl X-Patchwork-Id: 8615 Return-Path: Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) client-signature RSA-PSS (4096 bits)) (Client CN "mail01.haj.ipfire.org", Issuer "R10" (verified OK)) by web04.haj.ipfire.org (Postfix) with ESMTPS id 4ZfC1B3rv7z3x1X for ; Fri, 18 Apr 2025 11:18:06 +0000 (UTC) Received: from mail02.haj.ipfire.org (mail02.haj.ipfire.org [172.28.1.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) client-signature ECDSA (secp384r1)) (Client CN "mail02.haj.ipfire.org", Issuer "E5" (verified OK)) by mail01.ipfire.org (Postfix) with ESMTPS id 4ZfC1860tJz3Nx for ; Fri, 18 Apr 2025 11:18:04 +0000 (UTC) Received: from mail02.haj.ipfire.org (localhost [127.0.0.1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4ZfC175YGfz337h for ; Fri, 18 Apr 2025 11:18:03 +0000 (UTC) X-Original-To: development@lists.ipfire.org Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) client-signature RSA-PSS (4096 bits)) (Client CN "mail01.haj.ipfire.org", Issuer "R10" (verified OK)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4ZfC145fnTz30Zp for ; Fri, 18 Apr 2025 11:18:00 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail01.ipfire.org (Postfix) with ESMTPSA id 4ZfC131F2yz35D; Fri, 18 Apr 2025 11:17:59 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003ed25519; t=1744975079; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=37j1i0ue6ZPaWOloxG+gTqRx+OwRTPyBfTVUAGj0w2U=; b=2irh7Iq9WlwKaZ/hSH9u49oE5Jzy1ODWVFPnAhJnbtqo+GRGbtQTe5+ff3AkXDAPg35bnr cfGWMT1DsqNZ/5Cw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003rsa; t=1744975079; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=37j1i0ue6ZPaWOloxG+gTqRx+OwRTPyBfTVUAGj0w2U=; b=H5acJBBDj3/BSen1jlplhjLiTmphjJRdNM4KguMl3z7sx/uSPUo7+BQA/L5EfRXxbXWF+s p9r93TTuL0irk3d+xLlMxQyxKZw6LuNzSyneO9rycHqY/nJZsQ/nPHYqy6aMiQYpONs2LM k/DI1c94gh/RWOMaYcPecgUww0PB8DPraFro1dPZmuoTP4caPHYALRwkL4k0hqNnh5wMHG twFU+CjXq2EZMqetCpBoP8uFyw6oyEDIOxaHSXotkiQ/rEthjosWpMR4vYiws1LFsqg2NB MJu00hgqVwaG610CVuFadkBDmLUKp0yr7fPZ/9XS/vfxOXcbUnSeznYKLrqj3w== From: Stefan Schantl To: development@lists.ipfire.org Cc: Stefan Schantl Subject: [PATCHv2 4/7] Move GetDyndnsRedIP from general-functions.pl to http-client-functions.pl Date: Fri, 18 Apr 2025 12:54:43 +0200 Message-ID: <20250418110741.7756-5-stefan.schantl@ipfire.org> In-Reply-To: <20250418110741.7756-1-stefan.schantl@ipfire.org> References: <20250418110741.7756-1-stefan.schantl@ipfire.org> Precedence: list List-Id: List-Subscribe: , List-Unsubscribe: , List-Post: List-Help: Sender: Mail-Followup-To: MIME-Version: 1.0 This function depends on the previously FetchPublicIp function and so also has to be moved. Signed-off-by: Stefan Schantl --- config/cfgroot/general-functions.pl | 25 --------------------- config/cfgroot/http-client-functions.pl | 30 +++++++++++++++++++++++++ 2 files changed, 30 insertions(+), 25 deletions(-) diff --git a/config/cfgroot/general-functions.pl b/config/cfgroot/general-functions.pl index 861f95dec..bbd0f9839 100644 --- a/config/cfgroot/general-functions.pl +++ b/config/cfgroot/general-functions.pl @@ -998,31 +998,6 @@ sub DyndnsServiceSync ($;$;$) { } return 0; } -# -# This sub returns the red IP used to compare in DyndnsServiceSync -# -sub GetDyndnsRedIP { - my %settings; - &General::readhash("${General::swroot}/ddns/settings", \%settings); - - open(IP, "${General::swroot}/red/local-ipaddress") or return 'unavailable'; - my $ip = ; - close(IP); - chomp $ip; - - # 100.64.0.0/10 is reserved for dual-stack lite (http://tools.ietf.org/html/rfc6598). - if (&General::IpInSubnet ($ip,'10.0.0.0','255.0.0.0') || - &General::IpInSubnet ($ip,'172.16.0.0.','255.240.0.0') || - &General::IpInSubnet ($ip,'192.168.0.0','255.255.0.0') || - &General::IpInSubnet ($ip,'100.64.0.0', '255.192.0.0')) - { - if ($settings{'BEHINDROUTER'} eq 'FETCH_IP') { - my $RealIP = &General::FetchPublicIp; - $ip = (&General::validip ($RealIP) ? $RealIP : 'unavailable'); - } - } - return $ip; -} # Translate ICMP code to text # ref: http://www.iana.org/assignments/icmp-parameters diff --git a/config/cfgroot/http-client-functions.pl b/config/cfgroot/http-client-functions.pl index bfb9fdd20..f1f7de309 100644 --- a/config/cfgroot/http-client-functions.pl +++ b/config/cfgroot/http-client-functions.pl @@ -307,4 +307,34 @@ sub FetchPublicIp { return; } +# +# This sub returns the red IP used to compare in DyndnsServiceSync +# +sub GetDyndnsRedIP { + my %settings; + + # Read-in ddns settings. + &General::readhash("${General::swroot}/ddns/settings", \%settings); + + # Try to grab the address from the local-ipaddress file. + my $ip = &General::grab_address_from_file("${General::swroot}/red/local-ipaddress") or return 'unavailable'; + + # 100.64.0.0/10 is reserved for dual-stack lite (http://tools.ietf.org/html/rfc6598). + if (&General::IpInSubnet ($ip,'10.0.0.0','255.0.0.0') || + &General::IpInSubnet ($ip,'172.16.0.0.','255.240.0.0') || + &General::IpInSubnet ($ip,'192.168.0.0','255.255.0.0') || + &General::IpInSubnet ($ip,'100.64.0.0', '255.192.0.0')) + { + if ($settings{'BEHINDROUTER'} eq 'FETCH_IP') { + # Call function to omit the real address. + my $RealIP = &FetchPublicIp; + + # Check if the grabbed address is valid. + $ip = (&General::validip ($RealIP) ? $RealIP : 'unavailable'); + } + } + + return $ip; +} + 1; From patchwork Fri Apr 18 10:54:44 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Schantl X-Patchwork-Id: 8616 Return-Path: Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) client-signature RSA-PSS (4096 bits)) (Client CN "mail01.haj.ipfire.org", Issuer "R10" (verified OK)) by web04.haj.ipfire.org (Postfix) with ESMTPS id 4ZfC1D5Q4fz3wb5 for ; Fri, 18 Apr 2025 11:18:08 +0000 (UTC) Received: from mail02.haj.ipfire.org (mail02.haj.ipfire.org [172.28.1.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) client-signature ECDSA (secp384r1)) (Client CN "mail02.haj.ipfire.org", Issuer "E5" (verified OK)) by mail01.ipfire.org (Postfix) with ESMTPS id 4ZfC1C3Nmhz3Rv for ; Fri, 18 Apr 2025 11:18:07 +0000 (UTC) Received: from mail02.haj.ipfire.org (localhost [127.0.0.1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4ZfC195mfZz33g9 for ; Fri, 18 Apr 2025 11:18:05 +0000 (UTC) X-Original-To: development@lists.ipfire.org Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) client-signature RSA-PSS (4096 bits)) (Client CN "mail01.haj.ipfire.org", Issuer "R10" (verified OK)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4ZfC151YStz3348 for ; Fri, 18 Apr 2025 11:18:01 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail01.ipfire.org (Postfix) with ESMTPSA id 4ZfC134KdYz38X; Fri, 18 Apr 2025 11:17:59 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003ed25519; t=1744975079; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=h6z2RWSSN0OxK4mEJGGLqE35KhY6J8fYTdvjxI83Yg0=; b=p4mMWIXbumDT9r+TgYV2Bdg06oM726KFeXBDyTitDnEiOhgkCuhFAmrRcKPmZjXpX3afeQ D1k/wMPrPGVycLDQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003rsa; t=1744975079; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=h6z2RWSSN0OxK4mEJGGLqE35KhY6J8fYTdvjxI83Yg0=; b=totx9xXMS9rLzwqi6LfSLNchYnGbBmHHg6D20rGYu/5TK8bGdczgZ45ffooFw5f3VW3/qw bnjSDCQ1c7xfIy/yRQziRrMQ/qGO3VnFpd77X8ugPBt95mL7McsM7tWDIIPmF8P0u7nntT /mlYPT6SrOBRaMvQO/U7f8zFmAKfZYzD7WGHYV0BY6N3TVNNkhcZti6wmC4CVuFXdArx9S qH2j0J0dEujgkqHhjST9PGsgThiv3mQW1grb+FJia8Qm1yb3Tz5zM328bLWb/fuLcqtLY2 reYl8IDvsOWE0bL8lhvrnHXBS3QRN42tpvjD/bz23t8dNiJocgT3DZGRGEOVYg== From: Stefan Schantl To: development@lists.ipfire.org Cc: Stefan Schantl Subject: [PATCHv2 5/7] ddns.cgi, wio.cgi: Use GetDyndnsRedIP from http-client-functions.pl file Date: Fri, 18 Apr 2025 12:54:44 +0200 Message-ID: <20250418110741.7756-6-stefan.schantl@ipfire.org> In-Reply-To: <20250418110741.7756-1-stefan.schantl@ipfire.org> References: <20250418110741.7756-1-stefan.schantl@ipfire.org> Precedence: list List-Id: List-Subscribe: , List-Unsubscribe: , List-Post: List-Help: Sender: Mail-Followup-To: MIME-Version: 1.0 Signed-off-by: Stefan Schantl --- html/cgi-bin/ddns.cgi | 3 ++- html/cgi-bin/wio.cgi | 3 ++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/html/cgi-bin/ddns.cgi b/html/cgi-bin/ddns.cgi index 0e3ccbe45..34475b75c 100644 --- a/html/cgi-bin/ddns.cgi +++ b/html/cgi-bin/ddns.cgi @@ -29,6 +29,7 @@ use experimental 'smartmatch'; require '/var/ipfire/general-functions.pl'; require "${General::swroot}/lang.pl"; require "${General::swroot}/header.pl"; +require "${General::swroot}/http-client-functions.pl"; #workaround to suppress a warning when a variable is used only once my @dummy = ( ${Header::table2colour}, ${Header::colouryellow} ); @@ -559,7 +560,7 @@ open(FILE, $datafile) or die "Unable to open $datafile."; close(FILE); # Get IP address of the red interface. -my $ip = &General::GetDyndnsRedIP(); +my $ip = &HTTPClient::GetDyndnsRedIP(); my $id = 0; my $toggle_enabled; diff --git a/html/cgi-bin/wio.cgi b/html/cgi-bin/wio.cgi index f31f5d565..30a51104c 100644 --- a/html/cgi-bin/wio.cgi +++ b/html/cgi-bin/wio.cgi @@ -50,6 +50,7 @@ require '/var/ipfire/general-functions.pl'; require '/var/ipfire/network-functions.pl'; require '/var/ipfire/lang.pl'; require '/var/ipfire/header.pl'; +require '/var/ipfire/http-client-functions.pl'; require '/usr/lib/wio/wio-lib.pl'; require '/usr/lib/wio/wio-graphs.pl'; @@ -1163,7 +1164,7 @@ close (FILE); @temp = split (/\,/, $_); if ( $temp[7] eq "on" ) { - $bgcolor = ( &General::DyndnsServiceSync (&General::GetDyndnsRedIP,$temp[1],$temp[2]) ? "$Header::colourgreen" : "$Header::colourred" ); + $bgcolor = ( &General::DyndnsServiceSync (&HTTPClient::GetDyndnsRedIP,$temp[1],$temp[2]) ? "$Header::colourgreen" : "$Header::colourred" ); } else { $bgcolor = "blue"; From patchwork Fri Apr 18 10:54:45 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Schantl X-Patchwork-Id: 8617 Return-Path: Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) client-signature RSA-PSS (4096 bits)) (Client CN "mail01.haj.ipfire.org", Issuer "R10" (verified OK)) by web04.haj.ipfire.org (Postfix) with ESMTPS id 4ZfC1D5rR5z3x1X for ; Fri, 18 Apr 2025 11:18:08 +0000 (UTC) Received: from mail02.haj.ipfire.org (mail02.haj.ipfire.org [172.28.1.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) client-signature ECDSA (secp384r1)) (Client CN "mail02.haj.ipfire.org", Issuer "E5" (verified OK)) by mail01.ipfire.org (Postfix) with ESMTPS id 4ZfC1C4XFcz3D2 for ; Fri, 18 Apr 2025 11:18:07 +0000 (UTC) Received: from mail02.haj.ipfire.org (localhost [127.0.0.1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4ZfC196695z339d for ; Fri, 18 Apr 2025 11:18:05 +0000 (UTC) X-Original-To: development@lists.ipfire.org Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) client-signature RSA-PSS (4096 bits)) (Client CN "mail01.haj.ipfire.org", Issuer "R10" (verified OK)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4ZfC155JTnz338N for ; Fri, 18 Apr 2025 11:18:01 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail01.ipfire.org (Postfix) with ESMTPSA id 4ZfC136p9bz39L; Fri, 18 Apr 2025 11:17:59 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003ed25519; t=1744975080; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Y/f9Z/H4w4GTxg8Yoj3wbVK5L1mFygz5d3ZqE3GE+ws=; b=zKx8G0CZ8vvR6LofSqsMYXzcCIr9O2ExSOhmZkxXSgRUiG340COhVn12FHMdXEQbTqpE5K kqogzTH0fkcxURBA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003rsa; t=1744975080; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Y/f9Z/H4w4GTxg8Yoj3wbVK5L1mFygz5d3ZqE3GE+ws=; b=qz30em8FIDe9476Ygruxt8TrPfwoPYE/MpX/QApQqeSnMo5poODuqdUBrhxCqvVYHuyNJv sUo55Gc8SGczw4n8+Jnl55XgJyk55407x5o9G4q96KcAZBHVhaJZFWk/3T6x3sA2cJ06i/ AvaTjOv/hmt6BZ2r0X4dc4+q8TZkjbv2Q+DFf+bU4WHgc1X3i6xqwnJGzdHOtLCVBKid8Y VFmKgkHqoRIjCbDD3s6uCY6LGW6E0rfGW/6WRrkWvctuNpgMTxjcwP8rG1LITRoT64Tgvv flMZCLDE7emfSqa6Bg3KqAy8NR7E+fuJtWbw+XB9Xwp3y3B36zA3ZauBO4a/nw== From: Stefan Schantl To: development@lists.ipfire.org Cc: Stefan Schantl Subject: [PATCHv2 6/7] ids-functions.pl: Use new downloader function from http-client-functions.pl Date: Fri, 18 Apr 2025 12:54:45 +0200 Message-ID: <20250418110741.7756-7-stefan.schantl@ipfire.org> In-Reply-To: <20250418110741.7756-1-stefan.schantl@ipfire.org> References: <20250418110741.7756-1-stefan.schantl@ipfire.org> Precedence: list List-Id: List-Subscribe: , List-Unsubscribe: , List-Post: List-Help: Sender: Mail-Followup-To: MIME-Version: 1.0 Signed-off-by: Stefan Schantl --- config/cfgroot/ids-functions.pl | 206 +++----------------------------- 1 file changed, 18 insertions(+), 188 deletions(-) diff --git a/config/cfgroot/ids-functions.pl b/config/cfgroot/ids-functions.pl index 399f5cbf8..1a72e4c3e 100644 --- a/config/cfgroot/ids-functions.pl +++ b/config/cfgroot/ids-functions.pl @@ -27,6 +27,7 @@ package IDS; require '/var/ipfire/general-functions.pl'; require "${General::swroot}/network-functions.pl"; +require "${General::swroot}/http-client-functions.pl"; require "${General::swroot}/suricata/ruleset-sources"; # Load perl module to deal with Archives. @@ -44,15 +45,6 @@ use File::Path qw(rmtree); # Load module to get file stats. use File::stat; -# Load module to deal with temporary files. -use File::Temp; - -# Load module to deal with the date formats used by the HTTP protocol. -use HTTP::Date; - -# Load the libwwwperl User Agent module. -use LWP::UserAgent; - # Load function from posix module to format time strings. use POSIX qw (strftime); @@ -130,9 +122,6 @@ my $suricatactrl = "/usr/local/bin/suricatactrl"; # Prefix for each downloaded ruleset. my $dl_rulesfile_prefix = "idsrules"; -# Temporary directory to download the rules files. -my $tmp_dl_directory = "/var/tmp"; - # Temporary directory where the rulesets will be extracted. my $tmp_directory = "/tmp/ids_tmp"; @@ -299,61 +288,13 @@ sub checkdiskspace () { # ## This function is responsible for downloading the ruleset for a given provider. ## -## * At first it initialize the downloader and sets an upstream proxy if configured. -## * The next step will be to generate the final download url, by obtaining the URL for the desired -## ruleset and add the settings for the upstream proxy. -## * Finally the function will grab the rule file or tarball from the server. -## It tries to reduce the amount of download by using the "If-Modified-Since" HTTP header. -# -## Return codes: -## -## * "no url" - If no download URL could be gathered for the provider. -## * "not modified" - In case the already stored rules file is up to date. -## * "incomplete download" - When the remote file size differs from the downloaded file size. -## * "$error" - The error message generated from the LWP::User Agent module. +## It uses the LWP-based downloader function from the general-functions.pl to +## download the ruleset for a requested provider. # sub downloadruleset ($) { my ($provider) = @_; - # The amount of download attempts before giving up and - # logging an error. - my $max_dl_attempts = 3; - - # Read proxysettings. - my %proxysettings=(); - &General::readhash("${General::swroot}/proxy/settings", \%proxysettings); - - # Init the download module. - # - # Request SSL hostname verification and specify path - # to the CA file. - my $downloader = LWP::UserAgent->new( - ssl_opts => { - SSL_ca_file => '/etc/ssl/cert.pem', - verify_hostname => 1, - } - ); - - # Set timeout to 10 seconds. - $downloader->timeout(10); - - # Check if an upstream proxy is configured. - if ($proxysettings{'UPSTREAM_PROXY'}) { - my $proxy_url; - - $proxy_url = "http://"; - - # Check if the proxy requires authentication. - if (($proxysettings{'UPSTREAM_USER'}) && ($proxysettings{'UPSTREAM_PASSWORD'})) { - $proxy_url .= "$proxysettings{'UPSTREAM_USER'}\:$proxysettings{'UPSTREAM_PASSWORD'}\@"; - } - - # Add proxy server address and port. - $proxy_url .= $proxysettings{'UPSTREAM_PROXY'}; - - # Setup proxy settings. - $downloader->proxy(['http', 'https'], $proxy_url); - } + my %settings = (); # Grab the download url for the provider. my $url = $IDS::Ruleset::Providers{$provider}{'dl_url'}; @@ -371,141 +312,30 @@ sub downloadruleset ($) { # Abort and return "no url", if no url could be determined for the provider. return "no url" unless ($url); - # Pass the requested URL to the downloader. - my $request = HTTP::Request->new(GET => $url); - - # Generate temporary file name, located in the tempoary download directory and with a suffix of ".tmp". - # The downloaded file will be stored there until some sanity checks are performed. - my $tmp = File::Temp->new( SUFFIX => ".tmp", DIR => "$tmp_dl_directory/", UNLINK => 0 ); - my $tmpfile = $tmp->filename(); + # Pass the requested URL to the settings hash. + $settings{'URL'} = $url; # Call function to get the final path and filename for the downloaded file. my $dl_rulesfile = &_get_dl_rulesfile($provider); - # Check if the rulesfile already exits, because it has been downloaded in the past. - # - # In this case we are requesting the server if the remote file has been changed or not. - # This will be done by sending the modification time in a special HTTP header. - if (-f $dl_rulesfile) { - # Call stat on the file. - my $stat = stat($dl_rulesfile); - - # Omit the mtime of the existing file. - my $mtime = $stat->mtime; - - # Convert the timestamp into right format. - my $http_date = time2str($mtime); - - # Add the If-Modified-Since header to the request to ask the server if the - # file has been modified. - $request->header( 'If-Modified-Since' => "$http_date" ); - } - - # Read-in Etags file for known Etags if the file is present. - my %etags = (); - &General::readhash("$etags_file", \%etags) if (-f $etags_file); - - # Check if an Etag for the current provider is stored. - if ($etags{$provider}) { - # Grab the stored tag. - my $etag = $etags{$provider}; - - # Add an "If-None-Match header to the request to ask the server if the - # file has been modified. - $request->header( 'If-None-Match' => $etag ); - } - - my $dl_attempt = 1; - my $response; - - # Download and retry on failure. - while ($dl_attempt <= $max_dl_attempts) { - # Perform the request and save the output into the tmpfile. - $response = $downloader->request($request, $tmpfile); - - # Check if the download was successfull. - if($response->is_success) { - # Break loop. - last; - - # Check if the server responds with 304 (Not Modified). - } elsif ($response->code == 304) { - # Remove temporary file, if one exists. - unlink("$tmpfile") if (-e "$tmpfile"); - - # Return "not modified". - return "not modified"; - - # Check if we ran out of download re-tries. - } elsif ($dl_attempt eq $max_dl_attempts) { - # Obtain error. - my $error = $response->content; - - # Remove temporary file, if one exists. - unlink("$tmpfile") if (-e "$tmpfile"); - - # Return the error message from response.. - return "$error"; - } - - # Remove temporary file, if one exists. - unlink("$tmpfile") if (-e "$tmpfile"); + # Add the file information to the settings hash. + $settings{'FILE'} = $dl_rulesfile; - # Increase download attempt counter. - $dl_attempt++; - } - - # Obtain the connection headers. - my $headers = $response->headers; - - # Get the timestamp from header, when the file has been modified the - # last time. - my $last_modified = $headers->last_modified; - - # Get the remote size of the downloaded file. - my $remote_filesize = $headers->content_length; + # Add Etag details to the settings hash. + $settings{'ETAGSFILE'} = $etags_file; + $settings{'ETAGPREFIX'} = $provider; - # Grab the Etag from response it the server provides one. - if ($response->header('Etag')) { - # Add the Etag to the etags hash. - $etags{$provider} = $response->header('Etag'); + # Call the downloader and pass the settings hash. + my $response = &HTTPClient::downloader(%settings); - # Write the etags file. - &General::writehash($etags_file, \%etags); + # Return the response message if the downloader provided one. + if ($response) { + return $response; } - # Perform stat on the tmpfile. - my $stat = stat($tmpfile); - - # Grab the local filesize of the downloaded tarball. - my $local_filesize = $stat->size; - - # Check if both file sizes match. - if (($remote_filesize) && ($remote_filesize ne $local_filesize)) { - # Delete temporary file. - unlink("$tmpfile"); - - # Return "1" - false. - return "incomplete download"; - } - - # Overwrite the may existing rulefile or tarball with the downloaded one. - move("$tmpfile", "$dl_rulesfile"); - - # Check if we got a last-modified value from the server. - if ($last_modified) { - # Assign the last-modified timestamp as mtime to the - # rules file. - utime(time(), "$last_modified", "$dl_rulesfile"); - } - - # Delete temporary file. - unlink("$tmpfile"); - - # Set correct ownership for the tarball. - set_ownership("$dl_rulesfile"); + # Set correct ownership for the downloaded rules file. + &set_ownership("$dl_rulesfile"); - # If we got here, everything worked fine. Return nothing. return; } From patchwork Fri Apr 18 10:54:46 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Schantl X-Patchwork-Id: 8618 Return-Path: Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) client-signature RSA-PSS (4096 bits)) (Client CN "mail01.haj.ipfire.org", Issuer "R10" (verified OK)) by web04.haj.ipfire.org (Postfix) with ESMTPS id 4ZfC1D6ly7z3x3f for ; Fri, 18 Apr 2025 11:18:08 +0000 (UTC) Received: from mail02.haj.ipfire.org (mail02.haj.ipfire.org [172.28.1.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) client-signature ECDSA (secp384r1)) (Client CN "mail02.haj.ipfire.org", Issuer "E5" (verified OK)) by mail01.ipfire.org (Postfix) with ESMTPS id 4ZfC1C6LJpz36h for ; Fri, 18 Apr 2025 11:18:07 +0000 (UTC) Received: from mail02.haj.ipfire.org (localhost [127.0.0.1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4ZfC1B125Dz338P for ; Fri, 18 Apr 2025 11:18:06 +0000 (UTC) X-Original-To: development@lists.ipfire.org Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) client-signature RSA-PSS (4096 bits)) (Client CN "mail01.haj.ipfire.org", Issuer "R10" (verified OK)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4ZfC162PvRz339f for ; Fri, 18 Apr 2025 11:18:02 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail01.ipfire.org (Postfix) with ESMTPSA id 4ZfC142KKwz3G1; Fri, 18 Apr 2025 11:18:00 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003ed25519; t=1744975080; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=X2Mc5M1PkG8Fedb71c0+i82C23c9AvfA0ka8iKobqH8=; b=7PqgCi8NemDUxoDsezKCC+y6sQkeuMXLAkldGomOWpmNA2MiOAvF7ZDK5zunsGMeOceIor 1Owbyc7HSyZfmjAA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003rsa; t=1744975080; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=X2Mc5M1PkG8Fedb71c0+i82C23c9AvfA0ka8iKobqH8=; b=T+2bCSuNFnipgiE082cQ9d/ORykxmXzrzw5kjkPaKaNI/oRJ0sf/QnPP44aDXmuwGfdt2h dEEiPxSIEeRrQBTlusxc8tb+xJWViFLMnYksELj8eFK/jHx1QG6aLJUWUxhiPGHFaSDkEi 9/tHP0JgT1zKZFXd82WJ+loLoKKXCyYCOXL2EG9oHL3uTCppN4AK71zkNuxcquJl2UUnHW swdepjC9W00o/2J9lneBCCGZU89h9hOjUiMkPqJAjhJMJ+W7eHtsX+K7530ZPHd2GRkhQS xOWPBSKpaVQ5Ft5lxjx8mFo9CB7yEISbVHdZvE9vZ2LXpCu1x0k5Gdskkd2H0g== From: Stefan Schantl To: development@lists.ipfire.org Cc: Stefan Schantl Subject: [PATCHv2 7/7] http-client-functions.pl: Allow to user define the timeout value. Date: Fri, 18 Apr 2025 12:54:46 +0200 Message-ID: <20250418110741.7756-8-stefan.schantl@ipfire.org> In-Reply-To: <20250418110741.7756-1-stefan.schantl@ipfire.org> References: <20250418110741.7756-1-stefan.schantl@ipfire.org> Precedence: list List-Id: List-Subscribe: , List-Unsubscribe: , List-Post: List-Help: Sender: Mail-Followup-To: MIME-Version: 1.0 This allows to specify the the timeout value. Defaults to to 60 seconds if not set. Signed-off-by: Stefan Schantl --- config/cfgroot/http-client-functions.pl | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/config/cfgroot/http-client-functions.pl b/config/cfgroot/http-client-functions.pl index f1f7de309..c9484c575 100644 --- a/config/cfgroot/http-client-functions.pl +++ b/config/cfgroot/http-client-functions.pl @@ -89,6 +89,10 @@ sub downloader (%) { $etagprefix = $args{"ETAGPREFIX"} if (exists($args{"ETAGPREFIX"})); my $max_size = $args{"MAXSIZE"} if (exists($args{"MAXSIZE"})); + # Timeout defaults to 60 Seconds if not set. + my $timeout = 60; + $timeout = $args{"TIMEOUT"} if (exists($args{"TIMEOUT"})); + # Abort with error "no url", if no URL has been given. die "downloader: No URL has been given." unless ($url); @@ -110,8 +114,9 @@ sub downloader (%) { }, ); - # Set timeout to 10 seconds. - $ua->timeout(10); + # Set the timeout to the configured value. + # Defaults to 60 seconds if not set. + $ua->timeout($timeout); # Assign maximum download size if set. $ua->max_size($max_size) if ($max_size);