[1/3] rrdimage: Add scripts for new graph display method

Message ID 20210401133516.1058-1-hofmann@leo-andres.de
State Accepted
Commit 910f1e8494a0c5bc323feb100a3666ed857fa0d3
Headers
Series [1/3] rrdimage: Add scripts for new graph display method |

Commit Message

Leo-Andres Hofmann April 1, 2021, 1:35 p.m. UTC
  This patch adds two scripts which will later be used to display graphs:

-> getrrdimage.cgi: Generates PNG images for graphs.
Until now, each CGI with embedded graphs had to be able to output
images. These functions are now gathered in this new script.
The additional parameter handling can be removed and the CGIs can
be simplified. This makes it easier to use and output the graphs.

-> rrdimage.js: Interactive Javascript functions
This allows the user to select time ranges without reloading the page.
In addition, the graphs are now periodically updated, allowing users
to live monitor the data.

Signed-off-by: Leo-Andres Hofmann <hofmann@leo-andres.de>
---
 config/cfgroot/graphs.pl                   |  10 +-
 config/rootfiles/common/web-user-interface |   2 +
 html/cgi-bin/getrrdimage.cgi               | 245 +++++++++++++++++++++
 html/html/include/rrdimage.js              | 122 ++++++++++
 4 files changed, 377 insertions(+), 2 deletions(-)
 create mode 100644 html/cgi-bin/getrrdimage.cgi
 create mode 100644 html/html/include/rrdimage.js
  

Comments

Leo-Andres Hofmann April 1, 2021, 1:36 p.m. UTC | #1
Hi all,

this series of patches is a follow-up to this discussion: https://lists.ipfire.org/pipermail/development/2021-March/009523.html

I decided to write a new CGI which handles the graph image generation (png). This allowed me to add new features and remove a lot of uneccessary parameter handling from the other CGIs.
I think this is in presentable form now. But there are still some issues I'd like to hear your opinion on:

- Because I have collected all the graph functions from the CGIs, there is now a very long if-elseif chain in getrrdimage.cgi. I tried my best to keep this as readable as possible. But I'd be happy to rewrite that if someone knows a better method.

- My test system doesn't have any hardware sensors or IPsec connections. I'm pretty sure that these graphs work but I can't test them.

- getrrdimage.cgi redirects graphs it can't generate to ensure compatibility with addons. Is that necessary?

Happy holidays!
Leo
  
Bernhard Bitsch April 1, 2021, 1:58 p.m. UTC | #2
Hi,

I didn't look at your patch in depth 'til now. But the integration of the errors from RRDtool looks good. Thanks for taking my suggestion in the discussion. I wasn't able to look at this since then, but it seems to be my solution I've tried some time ago on my system.
Could you test it with some errors of rrdgraph()?

Regards,
Bernhard

> Gesendet: Donnerstag, 01. April 2021 um 15:36 Uhr
> Von: "Leo Hofmann" <hofmann@leo-andres.de>
> An: development@lists.ipfire.org
> Betreff: Re: [PATCH 1/3] rrdimage: Add scripts for new graph display method
>
> Hi all,
>
> this series of patches is a follow-up to this discussion: https://lists.ipfire.org/pipermail/development/2021-March/009523.html
>
> I decided to write a new CGI which handles the graph image generation (png). This allowed me to add new features and remove a lot of uneccessary parameter handling from the other CGIs.
> I think this is in presentable form now. But there are still some issues I'd like to hear your opinion on:
>
> - Because I have collected all the graph functions from the CGIs, there is now a very long if-elseif chain in getrrdimage.cgi. I tried my best to keep this as readable as possible. But I'd be happy to rewrite that if someone knows a better method.
>
> - My test system doesn't have any hardware sensors or IPsec connections. I'm pretty sure that these graphs work but I can't test them.
>
> - getrrdimage.cgi redirects graphs it can't generate to ensure compatibility with addons. Is that necessary?
>
> Happy holidays!
> Leo
>
  
Leo-Andres Hofmann April 1, 2021, 2:18 p.m. UTC | #3
Hi Bernhard,

yes I liked your idea of having the error message directly visible in the diagram!
I deliberately put a typo in the RRD command. You can see the error message in the attached screenshot.

Best regards,
Leo

Am 01.04.2021 um 15:58 schrieb Bernhard Bitsch:
> Hi,
>
> I didn't look at your patch in depth 'til now. But the integration of the errors from RRDtool looks good. Thanks for taking my suggestion in the discussion. I wasn't able to look at this since then, but it seems to be my solution I've tried some time ago on my system.
> Could you test it with some errors of rrdgraph()?
>
> Regards,
> Bernhard
>
>> Gesendet: Donnerstag, 01. April 2021 um 15:36 Uhr
>> Von: "Leo Hofmann" <hofmann@leo-andres.de>
>> An: development@lists.ipfire.org
>> Betreff: Re: [PATCH 1/3] rrdimage: Add scripts for new graph display method
>>
>> Hi all,
>>
>> this series of patches is a follow-up to this discussion: https://lists.ipfire.org/pipermail/development/2021-March/009523.html
>>
>> I decided to write a new CGI which handles the graph image generation (png). This allowed me to add new features and remove a lot of uneccessary parameter handling from the other CGIs.
>> I think this is in presentable form now. But there are still some issues I'd like to hear your opinion on:
>>
>> - Because I have collected all the graph functions from the CGIs, there is now a very long if-elseif chain in getrrdimage.cgi. I tried my best to keep this as readable as possible. But I'd be happy to rewrite that if someone knows a better method.
>>
>> - My test system doesn't have any hardware sensors or IPsec connections. I'm pretty sure that these graphs work but I can't test them.
>>
>> - getrrdimage.cgi redirects graphs it can't generate to ensure compatibility with addons. Is that necessary?
>>
>> Happy holidays!
>> Leo
>>
  
Bernhard Bitsch April 1, 2021, 3:09 p.m. UTC | #4
Hi Leo,

yes that's exactly what I thought of.
The idea was inspired by post coming up sometimes in the community about empty graphs. Mostly just a guessing about the reason gives the solution more or less quickly/slow. Maybe the real error messages leads quicker to the issue.

Regards,
Bernhard

> Gesendet: Donnerstag, 01. April 2021 um 16:18 Uhr
> Von: "Leo Hofmann" <hofmann@leo-andres.de>
> An: "Bernhard Bitsch" <Bernhard.Bitsch@gmx.de>
> Cc: development@lists.ipfire.org
> Betreff: Re: [PATCH 1/3] rrdimage: Add scripts for new graph display method
>
> Hi Bernhard,
>
> yes I liked your idea of having the error message directly visible in the diagram!
> I deliberately put a typo in the RRD command. You can see the error message in the attached screenshot.
>
> Best regards,
> Leo
>
> Am 01.04.2021 um 15:58 schrieb Bernhard Bitsch:
> > Hi,
> >
> > I didn't look at your patch in depth 'til now. But the integration of the errors from RRDtool looks good. Thanks for taking my suggestion in the discussion. I wasn't able to look at this since then, but it seems to be my solution I've tried some time ago on my system.
> > Could you test it with some errors of rrdgraph()?
> >
> > Regards,
> > Bernhard
> >
> >> Gesendet: Donnerstag, 01. April 2021 um 15:36 Uhr
> >> Von: "Leo Hofmann" <hofmann@leo-andres.de>
> >> An: development@lists.ipfire.org
> >> Betreff: Re: [PATCH 1/3] rrdimage: Add scripts for new graph display method
> >>
> >> Hi all,
> >>
> >> this series of patches is a follow-up to this discussion: https://lists.ipfire.org/pipermail/development/2021-March/009523.html
> >>
> >> I decided to write a new CGI which handles the graph image generation (png). This allowed me to add new features and remove a lot of uneccessary parameter handling from the other CGIs.
> >> I think this is in presentable form now. But there are still some issues I'd like to hear your opinion on:
> >>
> >> - Because I have collected all the graph functions from the CGIs, there is now a very long if-elseif chain in getrrdimage.cgi. I tried my best to keep this as readable as possible. But I'd be happy to rewrite that if someone knows a better method.
> >>
> >> - My test system doesn't have any hardware sensors or IPsec connections. I'm pretty sure that these graphs work but I can't test them.
> >>
> >> - getrrdimage.cgi redirects graphs it can't generate to ensure compatibility with addons. Is that necessary?
> >>
> >> Happy holidays!
> >> Leo
> >>
>
  

Patch

diff --git a/config/cfgroot/graphs.pl b/config/cfgroot/graphs.pl
index e4c3613fb..beddff032 100644
--- a/config/cfgroot/graphs.pl
+++ b/config/cfgroot/graphs.pl
@@ -29,6 +29,12 @@  require '/var/ipfire/general-functions.pl';
 require "${General::swroot}/lang.pl";
 require "${General::swroot}/header.pl";
 
+# Graph image size in pixel
+our %image_size = ('width' => 910, 'height' => 300);
+
+# List of all available time ranges
+our @time_ranges = ("hour", "day", "week", "month", "year");
+
 my $ERROR;
 
 my @GRAPH_ARGS = (
@@ -48,8 +54,8 @@  my @GRAPH_ARGS = (
 	"-W www.ipfire.org",
 
 	# Default size
-	"-w 910",
-	"-h 300",
+	"-w $image_size{'width'}",
+	"-h $image_size{'height'}",
 
 	# Use alternative grid
 	"--alt-y-grid",
diff --git a/config/rootfiles/common/web-user-interface b/config/rootfiles/common/web-user-interface
index 540bf1e4b..23e9f3e5e 100644
--- a/config/rootfiles/common/web-user-interface
+++ b/config/rootfiles/common/web-user-interface
@@ -20,6 +20,7 @@  srv/web/ipfire/cgi-bin/extrahd.cgi
 srv/web/ipfire/cgi-bin/fireinfo.cgi
 srv/web/ipfire/cgi-bin/firewall.cgi
 srv/web/ipfire/cgi-bin/fwhosts.cgi
+srv/web/ipfire/cgi-bin/getrrdimage.cgi
 srv/web/ipfire/cgi-bin/gpl.cgi
 #srv/web/ipfire/cgi-bin/guardian.cgi
 srv/web/ipfire/cgi-bin/gui.cgi
@@ -300,6 +301,7 @@  srv/web/ipfire/html/images/view-refresh.png
 srv/web/ipfire/html/images/wakeup.gif
 srv/web/ipfire/html/images/window-new.png
 srv/web/ipfire/html/include
+srv/web/ipfire/html/include/rrdimage.js
 srv/web/ipfire/html/include/zoneconf.js
 srv/web/ipfire/html/index.cgi
 srv/web/ipfire/html/redirect-templates
diff --git a/html/cgi-bin/getrrdimage.cgi b/html/cgi-bin/getrrdimage.cgi
new file mode 100644
index 000000000..0caefe0ac
--- /dev/null
+++ b/html/cgi-bin/getrrdimage.cgi
@@ -0,0 +1,245 @@ 
+#!/usr/bin/perl
+###############################################################################
+#                                                                             #
+# IPFire.org - A linux based firewall                                         #
+# Copyright (C) 2005-2021  IPFire Team                                        #
+#                                                                             #
+# This program is free software: you can redistribute it and/or modify        #
+# it under the terms of the GNU General Public License as published by        #
+# the Free Software Foundation, either version 3 of the License, or           #
+# (at your option) any later version.                                         #
+#                                                                             #
+# This program is distributed in the hope that it will be useful,             #
+# but WITHOUT ANY WARRANTY; without even the implied warranty of              #
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the               #
+# GNU General Public License for more details.                                #
+#                                                                             #
+# You should have received a copy of the GNU General Public License           #
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.       #
+#                                                                             #
+###############################################################################
+
+use strict;
+use URI;
+use GD;
+use GD::Text::Wrap;
+use experimental 'smartmatch';
+
+# debugging
+#use warnings;
+#use CGI::Carp 'fatalsToBrowser';
+
+require '/var/ipfire/general-functions.pl';
+require "${General::swroot}/lang.pl";
+require "${General::swroot}/header.pl";
+require "${General::swroot}/graphs.pl";
+
+# List of graph origins that getrrdimage.cgi can process directly
+# (unknown origins are forwarded to ensure compatibility)
+my @supported_origins = ("entropy.cgi", "hardwaregraphs.cgi", "media.cgi",
+	"memory.cgi","netexternal.cgi", "netinternal.cgi", "netother.cgi",
+	"netovpnrw.cgi", "netovpnsrv.cgi", "qos.cgi", "system.cgi");
+
+### Process GET parameters ###
+# URL format: /?origin=[graph origin cgi]&graph=[graph name]&range=[time range]
+my $uri = URI->new($ENV{'REQUEST_URI'});
+my %query = $uri->query_form;
+
+my $origin = lc $query{'origin'}; # lower case
+my $graph = $query{'graph'};
+my $range = lc $query{'range'}; # lower case
+
+# Check parameters
+unless(($origin =~ /^\w+?\.cgi$/) && ($graph =~ /^[\w-]+?$/) && ($range ~~ @Graphs::time_ranges)) {
+	# Send HTTP headers
+	_start_png_output();
+	
+	_print_error("URL parameters missing or malformed.");	
+	exit;
+}
+
+# Unsupported graph origin: Redirect request to the CGI specified in the "origin" parameter
+# This enables backwards compatibility with addons that use Graphs::makegraphbox to ouput their own graphs
+unless($origin ~~ @supported_origins) {
+	# Rewrite to old URL format: /[graph origin cgi]?[graph name]?[time range]
+	my $location = "https://$ENV{'SERVER_NAME'}:$ENV{'SERVER_PORT'}/cgi-bin/${origin}?${graph}?${range}";
+	
+	# Send HTTP redirect
+	print "Status: 302 Found\n";
+	print "Location: $location\n";
+	print "Content-type: text/html; charset=UTF-8\n";
+	print "\n"; # End of HTTP headers
+	
+	print "Unsupported origin, request redirected to '$location'";
+	exit;
+}
+
+### Create graphs ###
+# Send HTTP headers
+_start_png_output();
+
+# Graphs are first grouped by their origin.
+# This is because some graph categories require special parameter handling.
+my $graphstatus = '';
+if($origin eq "entropy.cgi") {				## entropy.cgi
+	$graphstatus = Graphs::updateentropygraph($range);
+# ------
+
+} elsif($origin eq "hardwaregraphs.cgi") {	## hardwaregraphs.cgi
+	if($graph eq "hwtemp") {
+		$graphstatus = Graphs::updatehwtempgraph($range);
+	} elsif($graph eq "hwfan") {
+		$graphstatus = Graphs::updatehwfangraph($range);
+	} elsif($graph eq "hwvolt") {
+		$graphstatus = Graphs::updatehwvoltgraph($range);
+	} elsif($graph eq "thermaltemp") {
+		$graphstatus = Graphs::updatethermaltempgraph($range);
+	} elsif($graph =~ "sd?") {
+		$graphstatus = Graphs::updatehddgraph($graph, $range);
+	} elsif($graph =~ "nvme?") {
+		$graphstatus = Graphs::updatehddgraph($graph, $range);
+	} else {
+		$graphstatus = "Unknown graph name.";
+	}
+# ------
+
+} elsif($origin eq "media.cgi") {			## media.cgi
+	if ($graph =~ "sd?" || $graph =~ "mmcblk?" || $graph =~ "nvme?n?" || $graph =~ "xvd??" || $graph =~ "vd?" || $graph =~ "md*" ) {
+		$graphstatus = Graphs::updatediskgraph($graph, $range);
+	} else {
+		$graphstatus = "Unknown graph name.";
+	}
+# ------
+
+} elsif($origin eq "memory.cgi") {			## memory.cgi
+	if($graph eq "memory") {
+		$graphstatus = Graphs::updatememorygraph($range);
+	} elsif($graph eq "swap") {
+		$graphstatus = Graphs::updateswapgraph($range);
+	} else {
+		$graphstatus = "Unknown graph name.";
+	}
+# ------
+
+} elsif($origin eq "netexternal.cgi") {		## netexternal.cgi
+	$graphstatus = Graphs::updateifgraph($graph, $range);
+# ------
+
+} elsif($origin eq "netinternal.cgi") {		## netinternal.cgi
+	if ($graph =~ /wireless/){
+		$graph =~ s/wireless//g;
+		$graphstatus = Graphs::updatewirelessgraph($graph, $range);
+	} else {
+		$graphstatus = Graphs::updateifgraph($graph, $range);
+	}
+# ------
+
+} elsif($origin eq "netother.cgi") {		## netother.cgi
+	if($graph eq "conntrack") {
+		$graphstatus = Graphs::updateconntrackgraph($range);
+	} elsif($graph eq "fwhits") {
+		$graphstatus = Graphs::updatefwhitsgraph($range);
+	} else {
+		$graphstatus = Graphs::updatepinggraph($graph, $range);
+	}
+# ------
+
+} elsif($origin eq "netovpnrw.cgi") {		## netovpnrw.cgi
+	if($graph ne "UNDEF") {
+		$graphstatus = Graphs::updatevpngraph($graph, $range);
+	} else {
+		$graphstatus = "Unknown graph name.";
+	}
+# ------
+
+} elsif($origin eq "netovpnsrv.cgi") {		## netovpnsrv.cgi
+	if ($graph =~ /ipsec-/){
+		$graph =~ s/ipsec-//g;
+		$graphstatus = Graphs::updateifgraph($graph, $range);
+	} else {
+		$graphstatus = Graphs::updatevpnn2ngraph($graph, $range);
+	}
+# ------
+
+} elsif($origin eq "qos.cgi") { 			## qos.cgi
+	$graphstatus = Graphs::updateqosgraph($graph, $range);
+# ------
+
+} elsif($origin eq "services.cgi") {		## services.cgi
+	if($graph eq "processescpu") {
+		$graphstatus = Graphs::updateprocessescpugraph($range);
+	} elsif($graph eq "processesmemory") {
+		$graphstatus = Graphs::updateprocessesmemorygraph($range);
+	} else {
+		$graphstatus = "Unknown graph name.";
+	}
+# ------
+
+} elsif($origin eq "system.cgi") { 			## system.cgi
+	if($graph eq "cpu") {
+		$graphstatus = Graphs::updatecpugraph($range);
+	} elsif($graph eq "cpufreq") {
+		$graphstatus = Graphs::updatecpufreqgraph($range);
+	} elsif($graph eq "load") {
+		$graphstatus = Graphs::updateloadgraph($range);
+	} else {
+		$graphstatus = "Unknown graph name.";
+	}
+# ------
+
+} else {
+	$graphstatus = "Unknown graph origin.";
+}
+
+### Print error message ###
+# Add request parameters for debugging
+if($graphstatus) {
+	$graphstatus = "$graphstatus\n($origin, $graph, $range)";
+	_print_error($graphstatus);
+}
+
+###--- Internal functions ---###
+
+# Send HTTP headers and switch to binary output
+# (don't print any non-image data to STDOUT afterwards)
+sub _start_png_output {
+	print "Cache-Control: no-cache, no-store\n";
+	print "Content-Type: image/png\n";
+	print "\n"; # End of HTTP headers
+	binmode(STDOUT);
+}
+
+# Print error message to PNG output
+sub _print_error {
+	my ($message) = @_;
+	$message = "- Error -\n \n$message";
+
+	# Create new image with the same size as a graph
+	my $img = GD::Image->new($Graphs::image_size{'width'}, $Graphs::image_size{'height'});
+	$img->interlaced('true');
+
+	# Basic colors
+	my $color_background = $img->colorAllocate(255, 255, 255);
+	my $color_border = $img->colorAllocate(255, 0, 0);
+	my $color_text = $img->colorAllocate(0, 0, 0);
+
+	# Background and border
+	$img->setThickness(2);
+	$img->filledRectangle(0, 0, $img->width, $img->height, $color_background);
+	$img->rectangle(10, 10, $img->width - 10, $img->height - 10, $color_border);
+	
+	# Draw message with line-wrap
+	my $textbox = GD::Text::Wrap->new($img,
+		text => $message,
+		width => ($img->width - 50),
+		color => $color_text,
+		align => 'center',
+		line_space => 5,
+		preserve_nl => 1
+	);
+	$textbox->set_font(gdLargeFont);
+	$textbox->draw(25, 25);
+
+	# Get PNG output
+	print $img->png;
+}
diff --git a/html/html/include/rrdimage.js b/html/html/include/rrdimage.js
new file mode 100644
index 000000000..e7ee4c769
--- /dev/null
+++ b/html/html/include/rrdimage.js
@@ -0,0 +1,122 @@ 
+/*#############################################################################
+#                                                                             #
+# IPFire.org - A linux based firewall                                         #
+# Copyright (C) 2007-2021  IPFire Team  <info@ipfire.org>                     #
+#                                                                             #
+# This program is free software: you can redistribute it and/or modify        #
+# it under the terms of the GNU General Public License as published by        #
+# the Free Software Foundation, either version 3 of the License, or           #
+# (at your option) any later version.                                         #
+#                                                                             #
+# This program is distributed in the hope that it will be useful,             #
+# but WITHOUT ANY WARRANTY; without even the implied warranty of              #
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the               #
+# GNU General Public License for more details.                                #
+#                                                                             #
+# You should have received a copy of the GNU General Public License           #
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.       #
+#                                                                             #
+#############################################################################*/
+
+// "onclick" event handler for graph time range select button
+// buttonObj: reference to the button
+function rrdimage_selectRange(buttonObj) {
+	if(! (buttonObj && ('range' in buttonObj.dataset))) {
+		return; //required parameters are missing
+	}
+
+	// Get selected time range from button
+	const range = buttonObj.dataset.range;
+	
+	// Get surrounding div box and select new range 
+	let graphBox = $(buttonObj).closest('div');
+	_rrdimg_setRange(graphBox, range);
+}
+
+// Document loaded: Process all graphs, start reload timers
+$(function() {
+	$('div.rrdimage').each(function() {
+		let graphBox = $(this);
+		_rrdimg_setRange(graphBox, graphBox.data('defaultRange'), true);
+	});
+});
+
+//--- Internal functions ---
+
+// Set or update graph time range, start automatic reloading
+// graphBox: jQuery object, reference to graph div box
+// range: time range (day, hour, ...)
+// initMode: don't immediately reload graph, but force timers and attributes update
+function _rrdimg_setRange(graphBox, range, initMode = false) {
+	if(! ((graphBox instanceof jQuery) && (graphBox.length === 1))) {
+		return; //graphBox element missing
+	}
+
+	// Check range parameter, default to "day" on error
+	if(! ["hour", "day", "week", "month", "year"].includes(range)) {
+		range = "day";
+	}
+
+	// Check if the time range is changed
+	if((graphBox.data('range') !== range) || initMode) {
+		graphBox.data('range', range); //Store new range
+		
+		// Update button highlighting
+		graphBox.find('button').removeClass('selected');
+		graphBox.find(`button[data-range="${range}"]`).addClass('selected');
+	}
+
+	// Clear pending reload timer to prevent multiple image reloads
+	let timerId = graphBox.data('reloadTimer');
+	if(timerId !== undefined) {
+		window.clearInterval(timerId);
+		graphBox.removeData('reloadTimer');
+	}
+
+	// Determine auto reload interval (in seconds),
+	// interval = 0 disables auto reloading by default
+	let interval = 0;
+	switch(range) {
+		case 'hour':
+			interval = 60;
+			break;
+
+		case 'day':
+		case 'week':
+			interval = 300;
+			break;
+	}
+
+	// Start reload timer and store reference
+	if(interval > 0) {
+		timerId = window.setInterval(function(graphRef) {
+			_rrdimg_reload(graphRef);
+		}, interval * 1000, graphBox);
+		graphBox.data('reloadTimer', timerId);
+	}
+
+	// Always reload image unless disabled by init mode
+	if(! initMode) {
+		_rrdimg_reload(graphBox);
+	}
+}
+
+// Reload graph image, add timestamp to prevent caching
+// graphBox: jQuery object (graph element must be valid)
+function _rrdimg_reload(graphBox) {
+	const origin = graphBox.data('origin');
+	const graph = graphBox.data('graph');	
+	const timestamp = Date.now();
+
+	// Get user selected range or fall back to default
+	let range = graphBox.data('range');
+	if(! range) {
+		range = graphBox.data('defaultRange');
+	}
+
+	// Generate new image URL with timestamp
+	const imageUrl = `/cgi-bin/getrrdimage.cgi?origin=${origin}&graph=${graph}&range=${range}&timestamp=${timestamp}`;
+
+	// Get graph image and set new URL
+	graphBox.children('img').first().attr('src', imageUrl);
+}