From patchwork Sat Mar 27 21:14:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adolf Belka X-Patchwork-Id: 3987 Return-Path: Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by web04.haj.ipfire.org (Postfix) with ESMTPS id 4F7BSM2qsQz3wjg for ; Sat, 27 Mar 2021 21:14:59 +0000 (UTC) Received: from mail02.haj.ipfire.org (mail02.haj.ipfire.org [172.28.1.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384 client-signature ECDSA (P-384) client-digest SHA384) (Client CN "mail02.haj.ipfire.org", Issuer "R3" (verified OK)) by mail01.ipfire.org (Postfix) with ESMTPS id 4F7BSL0jyzzQ1; Sat, 27 Mar 2021 21:14:58 +0000 (UTC) Received: from mail02.haj.ipfire.org (localhost [127.0.0.1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4F7BSK4lv3z2y51; Sat, 27 Mar 2021 21:14:57 +0000 (UTC) Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4F7BSH61S2z2xq4 for ; Sat, 27 Mar 2021 21:14:55 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by mail01.ipfire.org (Postfix) with ESMTPSA id 4F7BSG5BlPzQ1; Sat, 27 Mar 2021 21:14:54 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003ed25519; t=1616879694; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=BxJlIUhnS0a31VR2pY5SqRqW6XBAq03NuDPbsJSm9eQ=; b=EMhDfPcyZnTg2PCFp9Ve2PNQE6H2d5SzkJjtv2BwIsvJiGsifDRjXr3JMUiylNpubHp3Ta Y0u7aWtiXHu4YYBQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003rsa; t=1616879694; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=BxJlIUhnS0a31VR2pY5SqRqW6XBAq03NuDPbsJSm9eQ=; b=CZmnmujUeQ37HHElVnp2CWN6T2hOQpKRMbBDtv6S7Djgd5XG4nNHwgCsIpSQKv43MraxQd tFZ2m/InLcBfZb5Qyq01KozqEX9qnDdRUt2Ao0AE+48G5Md80ASM4xbnxP0dJ5+iCXEbDk vlxhitInXpk4YHS18Ghc8VWVrha98Gtz+WFu4nXDx44DP4nHOxeqyweVfBs9iaKDsjbOEG 2IitKOY01vFeFH8kX111FLvN1zCDRYFfbFquyd5MuqK6cFfN25787N2nOlFS1Z8uV0oYMi LE/z84q0R4uiIdZ06+h77eCS47Z5lgcr69GQ1euBYCBK/TSmsTuaVL5/Epy6/A== From: Adolf Belka To: development@lists.ipfire.org Subject: [PATCH v2 1/2] pmacct: New addon Date: Sat, 27 Mar 2021 22:14:49 +0100 Message-Id: <20210327211450.7090-1-adolf.belka@ipfire.org> MIME-Version: 1.0 X-BeenThere: development@lists.ipfire.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: IPFire development talk List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jon Murphy Errors-To: development-bounces@lists.ipfire.org Sender: "Development" - What is it? pmacct is a monitoring tool for network management tasks. Data collected can be used for analysis and troubleshooting purposes to maintain the health of the network. pmacct can collect, replicate and export network information. It can cache in memory tables, store persistently to SQLite3 and output to flat-files like CSV, formatted, and JSON. - Why is it needed? To monitor data usage (IP-based or MAC-based data accounting) down to the client level. Net-Traffic will monitor traffic for the entire RED, GREEN, etc. networks, but it cannot pinpoint which client is using lots of data. Connections will take a snapshot but not show day by day sums. pmacct can help admins keep tabs on users that use too much data. - What are the use cases? An ISP may implement data caps and if the limit is over-run then you have to pay for every additional xxGB of data used. Typical charges can be around $10 per 50GB. With pmacct you can identify the high users and take action, hopefully before the limit is breached. - This is being introduced as a command line only tool. However, at a later date, if it is useful to enough additional users a WUI page could be developed as discussed in the development mailing list https://lists.ipfire.org/pipermail/development/2021-January/009174.html - Changes in V2 version - Initscript is using IPFire template and installed with IPFire method. - All other daemons except pmacct and pmacctd have been removed from the install. - Example conf files have been removed from /etc/pmacct Both example conf files are described in the pmacct wiki draft. Tested-by: Jon Murphy Signed-off-by: Jon Murphy Signed-off-by: Erik Kapfer Signed-off-by: Adolf Belka --- config/backup/includes/pmacct | 2 + config/pmacct/pmacct.conf | 24 +++++++ config/rootfiles/packages/pmacct | 109 +++++++++++++++++++++++++++++++ lfs/pmacct | 100 ++++++++++++++++++++++++++++ make.sh | 2 + src/initscripts/packages/pmacct | 33 ++++++++++ src/paks/pmacct/install.sh | 36 ++++++++++ src/paks/pmacct/uninstall.sh | 33 ++++++++++ src/paks/pmacct/update.sh | 26 ++++++++ 9 files changed, 365 insertions(+) create mode 100644 config/backup/includes/pmacct create mode 100644 config/pmacct/pmacct.conf create mode 100644 config/rootfiles/packages/pmacct create mode 100644 lfs/pmacct create mode 100644 src/initscripts/packages/pmacct create mode 100755 src/paks/pmacct/install.sh create mode 100755 src/paks/pmacct/uninstall.sh create mode 100644 src/paks/pmacct/update.sh diff --git a/config/backup/includes/pmacct b/config/backup/includes/pmacct new file mode 100644 index 000000000..4359067ad --- /dev/null +++ b/config/backup/includes/pmacct @@ -0,0 +1,2 @@ +/etc/pmacct/ +/var/spool/pmacct/ diff --git a/config/pmacct/pmacct.conf b/config/pmacct/pmacct.conf new file mode 100644 index 000000000..3c1d47efa --- /dev/null +++ b/config/pmacct/pmacct.conf @@ -0,0 +1,24 @@ +! +! Pmacctd configuration file for IPFire environment +! + +syslog: daemon +daemonize: true +debug: false +promisc: true +interface: green0 + + +! +! "plugin1" plugin configuration +! +plugins: memory[plugin1] + +plugin_buffer_size[plugin1]: 102400 +plugin_pipe_size[plugin1]: 10240000 + +imt_mem_pools_number: 256 +imt_path[plugin1]: /var/spool/pmacct/plugin1.pipe + +aggregate[plugin1]: src_host, src_port, src_mac, dst_host, dst_port, dst_mac, proto +aggregate_filter[plugin1]: ip \ No newline at end of file diff --git a/config/rootfiles/packages/pmacct b/config/rootfiles/packages/pmacct new file mode 100644 index 000000000..e73a9b8a7 --- /dev/null +++ b/config/rootfiles/packages/pmacct @@ -0,0 +1,109 @@ +etc/pmacct +etc/pmacct/pmacct.conf +etc/rc.d/init.d/pmacct +usr/bin/pmacct +#usr/lib/pmacct +#usr/lib/pmacct/examples +#usr/lib/pmacct/examples/custom +#usr/lib/pmacct/examples/custom/libcustom.la +#usr/lib/pmacct/examples/lg +#usr/lib/pmacct/examples/lg/pmbgp +#usr/sbin/nfacctd +usr/sbin/pmacctd +#usr/sbin/pmbgpd +#usr/sbin/pmbmpd +#usr/sbin/pmtelemetryd +#usr/sbin/sfacctd +#usr/share/pmacct +#usr/share/pmacct/CONFIG-KEYS +#usr/share/pmacct/FAQS +#usr/share/pmacct/QUICKSTART +#usr/share/pmacct/UPGRADE +#usr/share/pmacct/docs +#usr/share/pmacct/docs/IPFIX +#usr/share/pmacct/docs/LOOKING_GLASS_FORMAT +#usr/share/pmacct/docs/MSGLOG_DUMP_FORMATS +#usr/share/pmacct/docs/SIGNALS +#usr/share/pmacct/docs/TRIGGER_VARS +#usr/share/pmacct/examples +usr/share/pmacct/examples/allow.lst.example +#usr/share/pmacct/examples/amqp +usr/share/pmacct/examples/amqp/amqp_receiver.py +#usr/share/pmacct/examples/avro +usr/share/pmacct/examples/avro/avro_file_decoder.py +usr/share/pmacct/examples/bgp_agent.map.example +usr/share/pmacct/examples/bgp_md5.lst.example +usr/share/pmacct/examples/bgp_xconnects.map.example +usr/share/pmacct/examples/flow_to_rd.map.example +#usr/share/pmacct/examples/kafka +usr/share/pmacct/examples/kafka/kafka_consumer.py +#usr/share/pmacct/examples/lg +usr/share/pmacct/examples/lg/pmbgp.py +usr/share/pmacct/examples/networks.lst.example +usr/share/pmacct/examples/nfacctd-sql.conf.example +usr/share/pmacct/examples/pcap_interfaces.map.example +usr/share/pmacct/examples/peers.map.example +usr/share/pmacct/examples/pmacctd-imt.conf.example +usr/share/pmacct/examples/pmacctd-multiple-plugins.conf.example +usr/share/pmacct/examples/pmacctd-sql.conf.example +usr/share/pmacct/examples/pmacctd-sqlite3.conf.example +usr/share/pmacct/examples/ports.lst.example +usr/share/pmacct/examples/pretag.map.example +usr/share/pmacct/examples/primitives.lst.example +usr/share/pmacct/examples/probe_netflow.conf.example +usr/share/pmacct/examples/probe_sflow.conf.example +usr/share/pmacct/examples/sampling.map.example +usr/share/pmacct/examples/tee_receivers.lst.example +#usr/share/pmacct/sql +#usr/share/pmacct/sql/README.64bit +#usr/share/pmacct/sql/README.GeoIP +#usr/share/pmacct/sql/README.IPv6 +#usr/share/pmacct/sql/README.cos +#usr/share/pmacct/sql/README.custom_primitives +#usr/share/pmacct/sql/README.etype +#usr/share/pmacct/sql/README.export_proto +#usr/share/pmacct/sql/README.iface +#usr/share/pmacct/sql/README.label +#usr/share/pmacct/sql/README.mask +#usr/share/pmacct/sql/README.mpls +#usr/share/pmacct/sql/README.mysql +#usr/share/pmacct/sql/README.nat +#usr/share/pmacct/sql/README.pgsql +#usr/share/pmacct/sql/README.sampling +usr/share/pmacct/sql/README.sqlite3 +#usr/share/pmacct/sql/README.tag2 +#usr/share/pmacct/sql/README.timestamp +#usr/share/pmacct/sql/README.tunnel +#usr/share/pmacct/sql/pmacct-create-db.pgsql +#usr/share/pmacct/sql/pmacct-create-db_bgp_v1.mysql +#usr/share/pmacct/sql/pmacct-create-db_v1.mysql +#usr/share/pmacct/sql/pmacct-create-db_v2.mysql +#usr/share/pmacct/sql/pmacct-create-db_v3.mysql +#usr/share/pmacct/sql/pmacct-create-db_v4.mysql +#usr/share/pmacct/sql/pmacct-create-db_v5.mysql +#usr/share/pmacct/sql/pmacct-create-db_v6.mysql +#usr/share/pmacct/sql/pmacct-create-db_v7.mysql +#usr/share/pmacct/sql/pmacct-create-db_v8.mysql +#usr/share/pmacct/sql/pmacct-create-db_v9.mysql +#usr/share/pmacct/sql/pmacct-create-table_bgp_v1.pgsql +usr/share/pmacct/sql/pmacct-create-table_bgp_v1.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v1.pgsql +usr/share/pmacct/sql/pmacct-create-table_v1.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v2.pgsql +usr/share/pmacct/sql/pmacct-create-table_v2.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v3.pgsql +usr/share/pmacct/sql/pmacct-create-table_v3.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v4.pgsql +usr/share/pmacct/sql/pmacct-create-table_v4.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v5.pgsql +usr/share/pmacct/sql/pmacct-create-table_v5.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v6.pgsql +usr/share/pmacct/sql/pmacct-create-table_v6.sqlite3 +usr/share/pmacct/sql/pmacct-create-table_v7.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v7_v8.pgsql +usr/share/pmacct/sql/pmacct-create-table_v8.sqlite3 +#usr/share/pmacct/sql/pmacct-create-table_v9.pgsql +usr/share/pmacct/sql/pmacct-create-table_v9.sqlite3 +#usr/share/pmacct/sql/pmacct-grant-db.mysql +var/ipfire/backup/addons/includes/pmacct +var/spool/pmacct diff --git a/lfs/pmacct b/lfs/pmacct new file mode 100644 index 000000000..30c6c47e1 --- /dev/null +++ b/lfs/pmacct @@ -0,0 +1,100 @@ +############################################################################### +# # +# IPFire.org - A linux based firewall # +# Copyright (C) 2019 IPFire Team # +# # +# This program is free software: you can redistribute it and/or modify # +# it under the terms of the GNU General Public License as published by # +# the Free Software Foundation, either version 3 of the License, or # +# (at your option) any later version. # +# # +# This program is distributed in the hope that it will be useful, # +# but WITHOUT ANY WARRANTY; without even the implied warranty of # +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # +# GNU General Public License for more details. # +# # +# You should have received a copy of the GNU General Public License # +# along with this program. If not, see . # +# # +############################################################################### + +############################################################################### +# Definitions +############################################################################### + +include Config + +VER = 1.7.6 + +THISAPP = pmacct-$(VER) +DL_FILE = $(THISAPP).tar.gz +DL_FROM = $(URL_IPFIRE) +DIR_APP = $(DIR_SRC)/$(THISAPP) +TARGET = $(DIR_INFO)/$(THISAPP) +PROG = pmacct +PAK_VER = 1 + +DEPS = libcdada + +############################################################################### +# Top-level Rules +############################################################################### + +objects = $(DL_FILE) + +$(DL_FILE) = $(DL_FROM)/$(DL_FILE) + +$(DL_FILE)_MD5 = b829cfdc394d5acd6d32fb23de181a6b + +install : $(TARGET) + +check : $(patsubst %,$(DIR_CHK)/%,$(objects)) + +download :$(patsubst %,$(DIR_DL)/%,$(objects)) + +md5 : $(subst %,%_MD5,$(objects)) + +dist: + @$(PAK) + +############################################################################### +# Downloading, checking, md5sum +############################################################################### + +$(patsubst %,$(DIR_CHK)/%,$(objects)) : + @$(CHECK) + +$(patsubst %,$(DIR_DL)/%,$(objects)) : + @$(LOAD) + +$(subst %,%_MD5,$(objects)) : + @$(MD5) + +############################################################################### +# Installation Details +############################################################################### + +$(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects)) + @$(PREBUILD) + @rm -rf $(DIR_APP) && cd $(DIR_SRC) && tar zxf $(DIR_DL)/$(DL_FILE) + cd $(DIR_APP) && ./autogen.sh + cd $(DIR_APP) && ./configure \ + --prefix=/usr \ + --sysconfdir=/etc/pmacct \ + --enable-sqlite3 \ + --enable-l2 \ + --enable-jansson + # Add directory for plugins and pipes + -mkdir -v /var/spool/pmacct + # Add configuration folder and copy conf file to it + -mkdir -v /etc/pmacct + cp -vf $(DIR_CONF)/pmacct/pmacct.conf /etc/pmacct + cd $(DIR_APP) && make $(MAKETUNING) + cd $(DIR_APP) && make install + #install initscripts + $(call INSTALL_INITSCRIPT,pmacct) + # Install backup definition + install -v -m 644 ${DIR_SRC}/config/backup/includes/pmacct \ + /var/ipfire/backup/addons/includes/pmacct + @rm -rf $(DIR_APP) + @$(POSTBUILD) diff --git a/make.sh b/make.sh index 0c124ac81..9b2b4f52e 100755 --- a/make.sh +++ b/make.sh @@ -1659,6 +1659,8 @@ buildipfire() { lfsmake2 ncdu lfsmake2 lshw lfsmake2 socat + lfsmake2 libcdada + lfsmake2 pmacct } buildinstaller() { diff --git a/src/initscripts/packages/pmacct b/src/initscripts/packages/pmacct new file mode 100644 index 000000000..45d4c336a --- /dev/null +++ b/src/initscripts/packages/pmacct @@ -0,0 +1,33 @@ +#!/bin/sh +# Begin $rc_base/init.d/pmacct + +# Based on sysklogd script from LFS-3.1 and earlier. + +. /etc/sysconfig/rc +. $rc_functions + +case "$1" in + start) + boot_mesg "Starting the pmacctd daemon..." + loadproc /usr/sbin/pmacctd -f /etc/pmacct/pmacct.conf + ;; + stop) + boot_mesg "Stopping the pmacctd daemon..." + killproc /usr/sbin/pmacctd + ;; + restart) + ${0} stop + sleep 1 + ${0} start + ;; + status) + statusproc /usr/sbin/pmacctd + ;; + *) + echo "Usage: $0 (start|stop|restart|status)" + exit 1 + ;; +esac + +# End $rc_base/init.d/pmacct + diff --git a/src/paks/pmacct/install.sh b/src/paks/pmacct/install.sh new file mode 100755 index 000000000..11b16f6c1 --- /dev/null +++ b/src/paks/pmacct/install.sh @@ -0,0 +1,36 @@ +#!/bin/bash +############################################################################ +# # +# This file is part of the IPFire Firewall. # +# # +# IPFire is free software; you can redistribute it and/or modify # +# it under the terms of the GNU General Public License as published by # +# the Free Software Foundation; either version 2 of the License, or # +# (at your option) any later version. # +# # +# IPFire is distributed in the hope that it will be useful, # +# but WITHOUT ANY WARRANTY; without even the implied warranty of # +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # +# GNU General Public License for more details. # +# # +# You should have received a copy of the GNU General Public License # +# along with IPFire; if not, write to the Free Software # +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # +# # +# Copyright (C) 2007 IPFire-Team . # +# # +############################################################################ +# +. /opt/pakfire/lib/functions.sh + +extract_files +restore_backup ${NAME} + +# Add symlinks for runlevels +ln -s ../init.d/${NAME} /etc/rc.d/rc0.d/K85${NAME} +ln -s ../init.d/${NAME} /etc/rc.d/rc3.d/S50${NAME} +ln -s ../init.d/${NAME} /etc/rc.d/rc6.d/K85${NAME} +start_service ${NAME} + + +# EOF diff --git a/src/paks/pmacct/uninstall.sh b/src/paks/pmacct/uninstall.sh new file mode 100755 index 000000000..51631e8c1 --- /dev/null +++ b/src/paks/pmacct/uninstall.sh @@ -0,0 +1,33 @@ +#!/bin/bash +############################################################################ +# # +# This file is part of the IPFire Firewall. # +# # +# IPFire is free software; you can redistribute it and/or modify # +# it under the terms of the GNU General Public License as published by # +# the Free Software Foundation; either version 2 of the License, or # +# (at your option) any later version. # +# # +# IPFire is distributed in the hope that it will be useful, # +# but WITHOUT ANY WARRANTY; without even the implied warranty of # +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # +# GNU General Public License for more details. # +# # +# You should have received a copy of the GNU General Public License # +# along with IPFire; if not, write to the Free Software # +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # +# # +# Copyright (C) 2007 IPFire-Team . # +# # +############################################################################ +# +. /opt/pakfire/lib/functions.sh + +stop_service ${NAME} +make_backup ${NAME} +remove_files + +# Delete symlinks +rm -rfv /etc/rc.d/rc?.d/???${NAME} + +# EOF diff --git a/src/paks/pmacct/update.sh b/src/paks/pmacct/update.sh new file mode 100644 index 000000000..89c40d0d7 --- /dev/null +++ b/src/paks/pmacct/update.sh @@ -0,0 +1,26 @@ +#!/bin/bash +############################################################################ +# # +# This file is part of the IPFire Firewall. # +# # +# IPFire is free software; you can redistribute it and/or modify # +# it under the terms of the GNU General Public License as published by # +# the Free Software Foundation; either version 2 of the License, or # +# (at your option) any later version. # +# # +# IPFire is distributed in the hope that it will be useful, # +# but WITHOUT ANY WARRANTY; without even the implied warranty of # +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # +# GNU General Public License for more details. # +# # +# You should have received a copy of the GNU General Public License # +# along with IPFire; if not, write to the Free Software # +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # +# # +# Copyright (C) 2007 IPFire-Team . # +# # +############################################################################ +# +. /opt/pakfire/lib/functions.sh +./uninstall.sh +./install.sh