From patchwork Thu Jul 30 19:15:52 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robin Roevens X-Patchwork-Id: 10080 Return-Path: Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (Client CN "mail01.haj.ipfire.org", Issuer "YR2" (not verified)) by web04.haj.ipfire.org (Postfix) with ESMTPS id 4hB0KB28m1z3wqJ for ; Thu, 30 Jul 2026 19:54:38 +0000 (UTC) Received: from mail02.haj.ipfire.org (mail02.haj.ipfire.org [IPv6:2001:678:b28::201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (Client CN "mail02.haj.ipfire.org", Issuer "YE1" (not verified)) by mail01.ipfire.org (Postfix) with ESMTPS id 4hB0K20qKWz2V for ; Thu, 30 Jul 2026 19:54:30 +0000 (UTC) Received: from mail02.haj.ipfire.org (localhost [IPv6:::1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4hB0GZ45W2z37BM for ; Thu, 30 Jul 2026 19:52:22 +0000 (UTC) X-Original-To: development@lists.ipfire.org Received: from mail01.ipfire.org (mail01.haj.ipfire.org [IPv6:2001:678:b28::25]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (Client CN "mail01.haj.ipfire.org", Issuer "YR2" (not verified)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4hB0GK1gfWz36WH for ; Thu, 30 Jul 2026 19:52:09 +0000 (UTC) Received: from layka.disroot.org (layka.disroot.org [178.21.23.139]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (Client did not present a certificate) by mail01.ipfire.org (Postfix) with ESMTPS id 4hB0G904XQz44j for ; Thu, 30 Jul 2026 19:52:00 +0000 (UTC) Authentication-Results: mail01.ipfire.org; dkim=pass header.d=disroot.org header.s=mail header.b=G4b32QcR; spf=pass (mail01.ipfire.org: domain of robin.roevens@disroot.org designates 178.21.23.139 as permitted sender) smtp.mailfrom=robin.roevens@disroot.org; dmarc=pass (policy=reject) header.from=disroot.org ARC-Seal: i=1; a=rsa-sha256; d=lists.ipfire.org; s=202003rsa; cv=none; t=1785441121; b=fjnb9K7sEKY0wDMX03/yhKMXt2oaCgJ5EX4/EGUmOIb/dizD7BBZWeMo0iVhwpDRv5a55H jsTrA5letqaIF+whA0YTjIhyw3av+3eIyHfCrbGTUncvAjAao3+68tTcoPazhV90bumfDV jYIRLwG2XE1iEBz2iy7y5+R+rFzlXylWsUb2oOLOy0rY+k4ULFs0EptSS/pa1CzDZ4JbfR Qyolns34IuoFWHmxDBV7aWkmzsAQbMT5WxX+P9WBkYncJfczMOCEyk+RIOmPUs+f58MYpQ /iKB0ivyJGAef9dcik0GU70SvEywtChNlsznzBkxEtoYmpQg6mzQLBmANPqoMw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=lists.ipfire.org; s=202003rsa; t=1785441121; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=k6QGlpT8SYjbpDIMY7vzyzvv39dOgyQz6sNVOdZld90=; b=wnZUudMnS/KzSdE6gWtsGNcA8b4J4WT9t9NbKxYTuk300+ZiVdIFH0B2kVjZRwD/FCGqBb k4F4IGwRnOkmfYl5jPQZsda41n7pDJSLL1o41ag0sWNh1B0fB7xiJVkKr6A5gX3ntTjgcU iqVOBsj3Ca71pRydiBznvO4JzPlNxfhzVoE8MopIGISj50Wd+LbmmjYTtexY6QQ7Po06gf 6UiOhN+7kLx/FmzyG/HPXODd+8t63O1gcNnn4vpnwWxBmoCTBpc9kAU8gEyReKNRBH70ce KMTu7e/mjjv1IAbhu5BvdByUWA512f1sTvnSrgNYljn1Ig8S75wyZ7Nvuv/cCQ== ARC-Authentication-Results: i=1; mail01.ipfire.org; dkim=pass header.d=disroot.org header.s=mail header.b=G4b32QcR; spf=pass (mail01.ipfire.org: domain of robin.roevens@disroot.org designates 178.21.23.139 as permitted sender) smtp.mailfrom=robin.roevens@disroot.org; dmarc=pass (policy=reject) header.from=disroot.org Received: from mail01.layka.lan (localhost [127.0.0.1]) by disroot.org (Postfix) with ESMTP id 9B74D41BD3 for ; Thu, 30 Jul 2026 21:52:00 +0200 (CEST) X-Virus-Scanned: SPAM Filter at disroot.org Received: from layka.disroot.org ([127.0.0.1]) by localhost (disroot.org [127.0.0.1]) (amavis, port 10024) with ESMTP id RrU967xW71KN for ; Thu, 30 Jul 2026 21:52:00 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=disroot.org; s=mail; t=1785441119; bh=BdreWelglfM5jtJ5dWS9JLfdBOnXMnBtTMhPeohRZ/0=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=G4b32QcReCU2dPsyNYirX2Gi0GVAxtNpxY6PfsBeJLZzX8nCxnlY8hJF4HybX8H5e 0HjmeQNVeedzlItvt3Q+K0FknSsscPCJ10D/sEb1fsm0WUJGBD8GQLH7PiCUS6M+Tw y52Llj+UmqnjmM9sEfq+DX/HNw8IzvEiAzyuoz5PUX1Aoaq+3J+l5qOqCALP9H1F0w i7l9uBsrtY6fiVOpK6tO7tbCn3RVm9n3Kj0txO5tbL0EqO41Ev7gtiv2dfK4qvAKrQ PIJYLOCAkjWNEeQ0HvJ+7WAAdMhaUdtZbf3gETYcK7SYaaz0pZY6nPvJbDZ8m/aKDe vUe1payMi3T8Q== Received: from chojin.roevenslambrechts.be (chojin.roevenslambrechts.be [192.168.0.50]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (no client certificate requested) (Authenticated sender) by hachiman (MailScanner Milter) with SMTP id ED4F4585FDA; Thu, 30 Jul 2026 21:51:53 +0200 (CEST) From: Robin Roevens To: development@lists.ipfire.org Cc: Robin Roevens Subject: [PATCH 1/5] Initialize async zabbix sender from zabbix_utils Date: Thu, 30 Jul 2026 21:15:52 +0200 Message-ID: <20260730195148.3278295-2-robin.roevens@disroot.org> In-Reply-To: <20260730195148.3278295-1-robin.roevens@disroot.org> References: <20260730195148.3278295-1-robin.roevens@disroot.org> Precedence: list List-Id: List-Subscribe: , List-Unsubscribe: , List-Post: List-Help: Sender: Mail-Followup-To: MIME-Version: 1.0 X-RoevensLambrechts-MailScanner-ID: ED4F4585FDA.AD543 X-RoevensLambrechts-MailScanner: Found to be clean X-RoevensLambrechts-MailScanner-From: robin.roevens@disroot.org X-RoevensLambrechts-MailScanner-Watermark: 1786045915.32049@4M0wFwCqx7fEdv+b8jQPvg X-Rspamd-Server: mail01.haj.ipfire.org X-Rspamd-Queue-Id: 4hB0G904XQz44j X-Rspamd-Action: no action X-Spamd-Result: default: False [-5.63 / 11.00]; BAYES_HAM(-3.00)[100.00%]; R_DKIM_ALLOW(-1.65)[disroot.org:s=mail]; MID_CONTAINS_FROM(1.00)[]; DKIM_REPUTATION(-0.92)[-0.92153870218341]; SPF_REPUTATION_HAM(-0.65)[-0.65402885146808]; DMARC_POLICY_ALLOW(-0.50)[disroot.org,reject]; R_MISSING_CHARSET(0.50)[]; R_SPF_ALLOW(-0.20)[+a:c]; MIME_GOOD(-0.10)[text/plain]; MX_GOOD(-0.10)[disroot.org]; RCPT_COUNT_TWO(0.00)[2]; ASN(0.00)[asn:50673, ipnet:178.21.23.0/24, country:NL]; IP_REPUTATION_HAM(0.00)[asn: 50673(0.00), country: NL(-0.01), ip: 178.21.23.139(0.00)]; ARC_NA(0.00)[]; TO_DN_SOME(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_THREE(0.00)[3]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MISSING_XM_UA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[disroot.org:+]; ARC_SIGNED(0.00)[lists.ipfire.org:s=202003rsa:i=1]; PREVIOUSLY_DELIVERED(0.00)[development@lists.ipfire.org]; FROM_HAS_DN(0.00)[] When Zabbix is enabled in new config section [zabbix], and the zabbix_utils python module is available, a zabbix AsyncSender object will be initialized for sending alerts to Zabbix using parameters from the new config section. Signed-off-by: Robin Roevens --- src/reporter.conf.in | 23 +++++++++++++++++++++++ src/suricata-reporter.in | 37 +++++++++++++++++++++++++++++++++++++ 2 files changed, 60 insertions(+) diff --git a/src/reporter.conf.in b/src/reporter.conf.in index 5943006..bab01b6 100644 --- a/src/reporter.conf.in +++ b/src/reporter.conf.in @@ -45,3 +45,26 @@ ; 3 = Low Severity ; 4 = Informational ;severity = 3 + +[zabbix] +; Enable sending alerts to Zabbix +;enabled = false + +; Path to the Zabbix agent configuration file +;zabbix_agentd_config = /etc/zabbix_agentd/zabbix_agentd.conf + +; Zabbix server ip or hostname (required if zabbix_agentd_config is not set) +;zabbix_server_host = 127.0.0.1 + +; Zabbix server port (defaults to 10051 if not set) +;zabbix_server_port = 10051 + +; Hostname as defined in Zabbix server to send alerts to (defaults to either the +; Hostname directive in Zabbix Agent config or system hostname) +;alert_item_hostname = IPFire + +; Zabbix item key to send alerts to +;alert_item_key = ipfire.suricata.event.get + +; Max age (seconds) to retry sending alerts to Zabbix +;alert_max_age = 3600 \ No newline at end of file diff --git a/src/suricata-reporter.in b/src/suricata-reporter.in index 28b55bc..f9da7b4 100644 --- a/src/suricata-reporter.in +++ b/src/suricata-reporter.in @@ -37,6 +37,13 @@ import socket import sqlite3 import sys +# Load zabbix_utils module if available +zabbix_utils_available = True +try: + from zabbix_utils import AsyncSender, ItemValue +except ImportError: + zabbix_utils_available = False + # Fetch the hostname HOSTNAME = socket.gethostname() @@ -75,6 +82,10 @@ class Reporter(object): # Remember the last time the database was cleaned self.last_cleanup_at = None + # Initialize Zabbix sender + self.zabbix_sender = None + self.init_zabbix_sender() + # Register any signals for signo in (signal.SIGINT, signal.SIGTERM): self.loop.add_signal_handler(signo, self.terminate) @@ -97,6 +108,32 @@ class Reporter(object): return config + def init_zabbix_sender(self): + """ + Initialize the Zabbix async sender if configured + """ + if not self.config.getboolean('zabbix', 'enabled', fallback=False): + return + + if not zabbix_utils_available: + log.error("zabbix-utils is not installed. Zabbix alerts will not be sent.") + return + + zabbix_config = self.config.get('zabbix', 'zabbix_agentd_config', fallback='') + zabbix_server_host = self.config.get('zabbix', 'zabbix_server_host', fallback='') + zabbix_server_port = self.config.getint('zabbix', 'zabbix_server_port', fallback=10051) + + if zabbix_config: + if not os.path.isfile(zabbix_config): + log.error(f"Zabbix agent config file {zabbix_config} does not exist.") + return + self.zabbix_sender = AsyncSender(use_config=True, config_path=zabbix_config) + else: + if not zabbix_server_host: + log.error("zabbix_server_host must be specified when zabbix_agentd_config is not provided.") + return + self.zabbix_sender = AsyncSender(server=zabbix_server_host, port=zabbix_server_port) + def _open_database(self): """ Opens the database From patchwork Thu Jul 30 19:15:53 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robin Roevens X-Patchwork-Id: 10079 Return-Path: Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (Client CN "mail01.haj.ipfire.org", Issuer "YR2" (not verified)) by web04.haj.ipfire.org (Postfix) with ESMTPS id 4hB0K434Kxz3wqJ for ; Thu, 30 Jul 2026 19:54:32 +0000 (UTC) Received: from mail02.haj.ipfire.org (mail02.haj.ipfire.org [IPv6:2001:678:b28::201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (Client CN "mail02.haj.ipfire.org", Issuer "YE1" (not verified)) by mail01.ipfire.org (Postfix) with ESMTPS id 4hB0Jw1ppCzyD for ; Thu, 30 Jul 2026 19:54:24 +0000 (UTC) Received: from mail02.haj.ipfire.org (localhost [IPv6:::1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4hB0GS75C4z37B3 for ; Thu, 30 Jul 2026 19:52:16 +0000 (UTC) X-Original-To: development@lists.ipfire.org Received: from mail01.ipfire.org (mail01.haj.ipfire.org [IPv6:2001:678:b28::25]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature ECDSA (secp384r1 raw public key) server-digest SHA384 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mail01.haj.ipfire.org", Issuer "YR2" (not verified)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4hB0GK0Zgsz36W7 for ; Thu, 30 Jul 2026 19:52:09 +0000 (UTC) Received: from layka.disroot.org (layka.disroot.org [178.21.23.139]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (Client did not present a certificate) by mail01.ipfire.org (Postfix) with ESMTPS id 4hB0G85Ndrz2l9 for ; Thu, 30 Jul 2026 19:52:00 +0000 (UTC) Authentication-Results: mail01.ipfire.org; dkim=pass header.d=disroot.org header.s=mail header.b=RF1b6yDR; spf=pass (mail01.ipfire.org: domain of robin.roevens@disroot.org designates 178.21.23.139 as permitted sender) smtp.mailfrom=robin.roevens@disroot.org; dmarc=pass (policy=reject) header.from=disroot.org ARC-Seal: i=1; a=rsa-sha256; d=lists.ipfire.org; s=202003rsa; cv=none; t=1785441120; b=BiDFZfOHL9CYFk6YOMX3RVusG0nJTJJHckS/CbUsvI5JqzuokKSvaGdInajqhSQwK/b/Qw 361kDL3IQUmxGNUxOqPCsx7QK3UgRaazk4Li8rdCpOCtwu9RW2vKFbubF2N2/ezy6uVqx4 TGh6tzYjQVroMxNHaxmvhtBvme3Y4EwqUfEOqOTxVfPT0unrcFdxWTwwVmpsN1UGE/madL kSPapFQjUVUT/rmb7XdBMvosK6c+sDxN+lwiMWejJ6POdueENGrV98ai1YwCTD5mvGlZLi 5w6txnR7e9fKEsmv4GEMX14zkIlw8g0guhxCeo8bgDF5SDniqZM9OI0xnfTuTg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=lists.ipfire.org; s=202003rsa; t=1785441120; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wa9UCLJmkFNw5yquMJuhLkMj2qi+T81GDifEheu+vZA=; b=Q+ndkf34HFyU4KnNFjtgHnYlhsZ9JMyMR/RIOSWt4R2Tp4gt5jJA+Ub2K4hph5kgX9x3HD 2e5UniLgolBdAn+eMM57Qk3qO1NeNowjJ+zlyELC1c2wmiVrImK6VzayZoKIxesQVJNjAe 0bamWgczOyJh/ZwSBoy6XOfP5QiRItQGnAkunWhFM4MeMh2CG60ud16oozd/vn1rn96vbK xavMDs1RcNeYdC+AdQsS5lK2FnuYosq4zBcpGitVECOBFDnBQly4jKNf3JgqEF0T8N/MT1 nugCzsJU9aM7J5zIYjmPuJqBP81RpSjpcpBs6ngNYHZBrcJ7+rLsLzTkutqy5Q== ARC-Authentication-Results: i=1; mail01.ipfire.org; dkim=pass header.d=disroot.org header.s=mail header.b=RF1b6yDR; spf=pass (mail01.ipfire.org: domain of robin.roevens@disroot.org designates 178.21.23.139 as permitted sender) smtp.mailfrom=robin.roevens@disroot.org; dmarc=pass (policy=reject) header.from=disroot.org Received: from mail01.layka.lan (localhost [127.0.0.1]) by disroot.org (Postfix) with ESMTP id 7148841C30 for ; Thu, 30 Jul 2026 21:52:00 +0200 (CEST) X-Virus-Scanned: SPAM Filter at disroot.org Received: from layka.disroot.org ([127.0.0.1]) by localhost (disroot.org [127.0.0.1]) (amavis, port 10024) with ESMTP id Za6amd6VHspQ for ; Thu, 30 Jul 2026 21:51:59 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=disroot.org; s=mail; t=1785441119; bh=3wsOZ3EaIg2Ff0uZDfFAgA/sXGNPhZfX6dLLZZ9H+Do=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=RF1b6yDRTWk1UY4GipNabLwFx5QCpgyiKP8PhRF9+B9SyOyViwlF4Gjoal2+yUEe4 SZaDgAzPR1Q+nDzuYFA6/LNWd1Tqnfr3zMa+YXRr4oHB0/d4fub5CIaOCEqUMfI4Ki foWRc1fAYTA1OCQPy8TyEQ5V8+dUqR2COaGxigNjaF6yWRlYOzbE5SUv7ADtPruE6k KGQeeQdOYC1BREMJuKYzbvJNt1ONLb3oG/q3P8vCIHqJ4iQcucSgu0sBAIUTICNIF2 m8kzL4ZdgALq8IStz+mt9vDLcckHkHhsb/K+HYKMmsCKz5rClY/kUJasYVSvoj+RAV yohUGH6e+UIMw== Received: from chojin.roevenslambrechts.be (chojin.roevenslambrechts.be [192.168.0.50]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (no client certificate requested) (Authenticated sender) by hachiman (MailScanner Milter) with SMTP id E433C585FDD; Thu, 30 Jul 2026 21:51:54 +0200 (CEST) From: Robin Roevens To: development@lists.ipfire.org Cc: Robin Roevens Subject: [PATCH 2/5] Add database column zabbix_pending in alerts table. Date: Thu, 30 Jul 2026 21:15:53 +0200 Message-ID: <20260730195148.3278295-3-robin.roevens@disroot.org> In-Reply-To: <20260730195148.3278295-1-robin.roevens@disroot.org> References: <20260730195148.3278295-1-robin.roevens@disroot.org> Precedence: list List-Id: List-Subscribe: , List-Unsubscribe: , List-Post: List-Help: Sender: Mail-Followup-To: MIME-Version: 1.0 X-RoevensLambrechts-MailScanner-ID: E433C585FDD.AD543 X-RoevensLambrechts-MailScanner: Found to be clean X-RoevensLambrechts-MailScanner-From: robin.roevens@disroot.org X-RoevensLambrechts-MailScanner-Watermark: 1786045915.34386@nO9qSAUCxwWA4OmOg4jCeA X-Rspamd-Server: mail01.haj.ipfire.org X-Rspamd-Queue-Id: 4hB0G85Ndrz2l9 X-Rspamd-Action: no action X-Spamd-Result: default: False [-5.63 / 11.00]; BAYES_HAM(-3.00)[100.00%]; R_DKIM_ALLOW(-1.65)[disroot.org:s=mail]; MID_CONTAINS_FROM(1.00)[]; DKIM_REPUTATION(-0.92)[-0.92153870218341]; SPF_REPUTATION_HAM(-0.65)[-0.65402885146808]; DMARC_POLICY_ALLOW(-0.50)[disroot.org,reject]; R_MISSING_CHARSET(0.50)[]; R_SPF_ALLOW(-0.20)[+a:c]; MIME_GOOD(-0.10)[text/plain]; MX_GOOD(-0.10)[disroot.org]; RCPT_COUNT_TWO(0.00)[2]; ASN(0.00)[asn:50673, ipnet:178.21.23.0/24, country:NL]; IP_REPUTATION_HAM(0.00)[asn: 50673(0.00), country: NL(-0.01), ip: 178.21.23.139(0.00)]; ARC_NA(0.00)[]; TO_DN_SOME(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_THREE(0.00)[3]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MISSING_XM_UA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[disroot.org:+]; ARC_SIGNED(0.00)[lists.ipfire.org:s=202003rsa:i=1]; PREVIOUSLY_DELIVERED(0.00)[development@lists.ipfire.org]; FROM_HAS_DN(0.00)[] To prevent possible hammering of the Zabbix server in busy environments, instead of sending alerts immediatly when they arrive, we will send them in bulk every 1 second. For that we need to know what alerts where not yet sent to Zabbix. This will be tracked in this new column zabbix_pending in the alerts table. Signed-off-by: Robin Roevens --- src/suricata-reporter.in | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/src/suricata-reporter.in b/src/suricata-reporter.in index f9da7b4..83e4e97 100644 --- a/src/suricata-reporter.in +++ b/src/suricata-reporter.in @@ -134,6 +134,27 @@ class Reporter(object): return self.zabbix_sender = AsyncSender(server=zabbix_server_host, port=zabbix_server_port) + def _ensure_zabbix_pending_column(self, db): + """ + Ensures the database schema always has the zabbix_pending column and index. + If they are missing, the database is migrated to add them. + """ + cursor = db.execute("PRAGMA table_info(alerts)") + columns = {row[1] for row in cursor.fetchall()} + + if "zabbix_pending" not in columns: + db.execute("ALTER TABLE alerts ADD COLUMN zabbix_pending INTEGER NOT NULL DEFAULT 0") + db.commit() + log.debug("Database: Added zabbix_pending column to alerts table.") + + cursor = db.execute("PRAGMA index_list(alerts)") + indexes = {row[1] for row in cursor.fetchall()} + + if "alerts_zabbix_pending" not in indexes: + db.execute("CREATE INDEX IF NOT EXISTS alerts_zabbix_pending ON alerts(zabbix_pending)") + db.commit() + log.debug("Database: Added alerts_zabbix_pending index on alerts table column zabbix_pending.") + def _open_database(self): """ Opens the database @@ -165,6 +186,8 @@ class Reporter(object): CREATE INDEX IF NOT EXISTS alerts_timestamp ON alerts(timestamp); """) + self._ensure_zabbix_pending_column(db) + return db @property From patchwork Thu Jul 30 19:15:54 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robin Roevens X-Patchwork-Id: 10077 Return-Path: Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature ECDSA (secp384r1 raw public key) server-digest SHA384 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mail01.haj.ipfire.org", Issuer "YR2" (not verified)) by web04.haj.ipfire.org (Postfix) with ESMTPS id 4hB0HR5XCTz3wqJ for ; Thu, 30 Jul 2026 19:53:07 +0000 (UTC) Received: from mail02.haj.ipfire.org (mail02.haj.ipfire.org [IPv6:2001:678:b28::201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (Client CN "mail02.haj.ipfire.org", Issuer "YE1" (not verified)) by mail01.ipfire.org (Postfix) with ESMTPS id 4hB0HF634rz47H for ; Thu, 30 Jul 2026 19:52:57 +0000 (UTC) Received: from mail02.haj.ipfire.org (localhost [IPv6:::1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4hB0GK3NSYz36Wk for ; Thu, 30 Jul 2026 19:52:09 +0000 (UTC) X-Original-To: development@lists.ipfire.org Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature ECDSA (secp384r1 raw public key) server-digest SHA384 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mail01.haj.ipfire.org", Issuer "YR2" (not verified)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4hB0GG6NF9z2xJ0 for ; Thu, 30 Jul 2026 19:52:06 +0000 (UTC) Received: from layka.disroot.org (layka.disroot.org [178.21.23.139]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (4096 bit raw public key) server-digest SHA256) (Client did not present a certificate) by mail01.ipfire.org (Postfix) with ESMTPS id 4hB0G64nKlz2V for ; Thu, 30 Jul 2026 19:51:58 +0000 (UTC) Authentication-Results: mail01.ipfire.org; dkim=pass header.d=disroot.org header.s=mail header.b=FOS0nepR; spf=pass (mail01.ipfire.org: domain of robin.roevens@disroot.org designates 178.21.23.139 as permitted sender) smtp.mailfrom=robin.roevens@disroot.org; dmarc=pass (policy=reject) header.from=disroot.org ARC-Seal: i=1; a=rsa-sha256; d=lists.ipfire.org; s=202003rsa; cv=none; t=1785441118; b=Kcf7PWuGR+W6oA3IU4HS5x2G795Rok8UdgTHPUAKCSndSJS0KoHiAzei4aC8Ncv8EhLqHP P5JULHQyEchc/ZLs6eqqa5yleqUU0+J/Zf+fdh0DjkfoamUMJH4YFDZoMRSrr7OtXS33zA 0RpWbdul1OPxjtOL9s2321ycsbGiCdpLSaOLCq9RdyWeXrt6Pa6WOkms3lyVk/8FOt80W6 wIXrkKdNhW5FeLVxNsflDKdJc1tKznSxH0CMSoGkRPuKjF2IKlI79ieIkiGnaP+jwuMfBZ 9FQ24pvWe2F6VMbUxguoRlHEI3TfvR88RYW31dx/vYeSWCLqa6XjC3hbieJLAw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=lists.ipfire.org; s=202003rsa; t=1785441118; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SE8Jk+gmJJFdc1+LmbaKUYv3LC8aiABQ403MrvEA5Wc=; b=Eve8QNoLrC7IwtpzqQRM9ioTfM0rVejHnskTi5Gb4WQAc/6zPtnD1T5k/VcfqbTlPcvlhM UNv/uhI+UnatcTMTYsKb7j3JTQc1cn6FzCgdKsBoEJAFM6DfEp9bO21o6gch0ZuPIxdiqB Fg6wfcQ3IlJYILW0rsdvoQ+2U36FiPXGPyRbW/XoBxcRbX4tXI/tfr1k3LCKXwUhqWURFU +gvOM6cr33wXcaUT8HAIvyqZKKvHtQgIW+2VlXYHhez4+UOWTVA5VqPcPfhp0coajb4z7Q U3D+J0OIm4x/E8aJEQUuWcY9iaNwWGPLVwIlENM3JDwq+/66wEjsQRfIH0LYaw== ARC-Authentication-Results: i=1; mail01.ipfire.org; dkim=pass header.d=disroot.org header.s=mail header.b=FOS0nepR; spf=pass (mail01.ipfire.org: domain of robin.roevens@disroot.org designates 178.21.23.139 as permitted sender) smtp.mailfrom=robin.roevens@disroot.org; dmarc=pass (policy=reject) header.from=disroot.org Received: from mail01.layka.lan (localhost [127.0.0.1]) by disroot.org (Postfix) with ESMTP id 12ABC41B43 for ; Thu, 30 Jul 2026 21:51:58 +0200 (CEST) X-Virus-Scanned: SPAM Filter at disroot.org Received: from layka.disroot.org ([127.0.0.1]) by localhost (disroot.org [127.0.0.1]) (amavis, port 10024) with ESMTP id vuV46lxNh64i for ; Thu, 30 Jul 2026 21:51:57 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=disroot.org; s=mail; t=1785441117; bh=wCjAXcjEHTM1CntbDAzR6ptpi/9ADPbh5PjTrkGyZgg=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=FOS0nepRXboMxhRyzibxJtZrZPYBPjNnn0lyIYjHVb+eUgX4018zouqWsxjEROBLU Q1VO4tUEMGT06KWhuSwai8veV/CGdfi6PXAowly0jy6Q0S7xkQEU6Lcd/+YFKFfNT1 h3L4NRbQDnVvVHMZkTuKcs0Dkc2jL1oQ98NGXXeOwiQbucdcDlPCh+57xln74xyF86 9IUYx3WL1+0hvC5V+lzxEytU7aoqW6IlNsqVwrNCofVi015rKtgHTkSxOb5JUQ2f+I MoEF39gNyuZsOQiRlpdH0tVVTuVrFt7prX/sA8mPqr+4MOgAak1yZyck7rlxm/aumO jmUtLVNnYNVmA== Received: from chojin.roevenslambrechts.be (chojin.roevenslambrechts.be [192.168.0.50]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (no client certificate requested) (Authenticated sender) by hachiman (MailScanner Milter) with SMTP id B36AE585FE0; Thu, 30 Jul 2026 21:51:54 +0200 (CEST) From: Robin Roevens To: development@lists.ipfire.org Cc: Robin Roevens Subject: [PATCH 3/5] Set zabbix_pending flag when storing new event in DB Date: Thu, 30 Jul 2026 21:15:54 +0200 Message-ID: <20260730195148.3278295-4-robin.roevens@disroot.org> In-Reply-To: <20260730195148.3278295-1-robin.roevens@disroot.org> References: <20260730195148.3278295-1-robin.roevens@disroot.org> Precedence: list List-Id: List-Subscribe: , List-Unsubscribe: , List-Post: List-Help: Sender: Mail-Followup-To: MIME-Version: 1.0 X-RoevensLambrechts-MailScanner-ID: B36AE585FE0.AD543 X-RoevensLambrechts-MailScanner: Found to be clean X-RoevensLambrechts-MailScanner-From: robin.roevens@disroot.org X-RoevensLambrechts-MailScanner-Watermark: 1786045915.98205@BLDBXlrhaWWAluiUOm3hAA X-Rspamd-Server: mail01.haj.ipfire.org X-Rspamd-Queue-Id: 4hB0G64nKlz2V X-Rspamd-Action: no action X-Spamd-Result: default: False [-5.63 / 11.00]; BAYES_HAM(-3.00)[100.00%]; R_DKIM_ALLOW(-1.65)[disroot.org:s=mail]; MID_CONTAINS_FROM(1.00)[]; DKIM_REPUTATION(-0.92)[-0.92153870218341]; SPF_REPUTATION_HAM(-0.65)[-0.65402885146808]; DMARC_POLICY_ALLOW(-0.50)[disroot.org,reject]; R_MISSING_CHARSET(0.50)[]; R_SPF_ALLOW(-0.20)[+a]; MIME_GOOD(-0.10)[text/plain]; MX_GOOD(-0.10)[disroot.org]; RCPT_COUNT_TWO(0.00)[2]; ASN(0.00)[asn:50673, ipnet:178.21.23.0/24, country:NL]; IP_REPUTATION_HAM(0.00)[asn: 50673(0.00), country: NL(-0.01), ip: 178.21.23.139(0.00)]; ARC_NA(0.00)[]; TO_DN_SOME(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_THREE(0.00)[3]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MISSING_XM_UA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[disroot.org:+]; ARC_SIGNED(0.00)[lists.ipfire.org:s=202003rsa:i=1]; PREVIOUSLY_DELIVERED(0.00)[development@lists.ipfire.org]; FROM_HAS_DN(0.00)[] When sending to zabbix is enabled, we need to set the zabbix_pending flag set on each new event written to the database. Signed-off-by: Robin Roevens --- src/suricata-reporter.in | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/src/suricata-reporter.in b/src/suricata-reporter.in index 83e4e97..78bb04d 100644 --- a/src/suricata-reporter.in +++ b/src/suricata-reporter.in @@ -314,9 +314,14 @@ class Reporter(object): """ Writes a single event to the database """ + # Determine whether this event should be marked for Zabbix delivery + zabbix_pending = 1 if self.config.getboolean('zabbix', 'enabled', fallback=False) else 0 + # Write the event to the database - self.db.execute("INSERT INTO alerts(timestamp, event) VALUES(?, ?)", - (event.timestamp.timestamp(), event.json)) + self.db.execute( + "INSERT INTO alerts(timestamp, event, zabbix_pending) VALUES(?, ?, ?)", + (event.timestamp.timestamp(), event.json, zabbix_pending) + ) # Commit it straight away self.db.commit() From patchwork Thu Jul 30 19:15:55 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robin Roevens X-Patchwork-Id: 10078 Return-Path: Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (Client CN "mail01.haj.ipfire.org", Issuer "YR2" (not verified)) by web04.haj.ipfire.org (Postfix) with ESMTPS id 4hB0J50TdWz3wqJ for ; Thu, 30 Jul 2026 19:53:41 +0000 (UTC) Received: from mail02.haj.ipfire.org (mail02.haj.ipfire.org [172.28.1.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (Client CN "mail02.haj.ipfire.org", Issuer "YE1" (not verified)) by mail01.ipfire.org (Postfix) with ESMTPS id 4hB0Hw6Gn7z6G0 for ; Thu, 30 Jul 2026 19:53:32 +0000 (UTC) Received: from mail02.haj.ipfire.org (localhost [IPv6:::1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4hB0GL2P6sz3771 for ; Thu, 30 Jul 2026 19:52:10 +0000 (UTC) X-Original-To: development@lists.ipfire.org Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (Client CN "mail01.haj.ipfire.org", Issuer "YR2" (not verified)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4hB0GH5dHHz33bJ for ; Thu, 30 Jul 2026 19:52:07 +0000 (UTC) Received: from layka.disroot.org (layka.disroot.org [178.21.23.139]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (Client did not present a certificate) by mail01.ipfire.org (Postfix) with ESMTPS id 4hB0G74gfkz2h2 for ; Thu, 30 Jul 2026 19:51:59 +0000 (UTC) Authentication-Results: mail01.ipfire.org; dkim=pass header.d=disroot.org header.s=mail header.b=M4wzHVJp; spf=pass (mail01.ipfire.org: domain of robin.roevens@disroot.org designates 178.21.23.139 as permitted sender) smtp.mailfrom=robin.roevens@disroot.org; dmarc=pass (policy=reject) header.from=disroot.org ARC-Seal: i=1; a=rsa-sha256; d=lists.ipfire.org; s=202003rsa; cv=none; t=1785441119; b=mmVfG3Iqh77NmSppT6lqVzwZMM/woRFbLdyb7UtnsVTnRcodZ45iXCVQGwMFMmAKEoJiqL M0LtcwkdPHLGvo38eb2jdw7IeDhScNzlcH1HlhpCCS9DMJmW7F1niaKviL60wRdobu2Ir9 VHfM3Cp/kgRMZTpyBIJ9WtapDM3kDT5FfaiOQu57/Sk/sclgKMyYFGJ9o4nklTPpx+oLkx cZJwnq7JtwsrcbOZDdc+RC8ijQGAVQEpqlAfYmymEz6G5LE0M/zHAOOK5+VrRKnPkBWnHg +15B2DqtkdfWl0mApre+F4ZZmFJLgfEGsNiG7w8QxeZ77Nk12SjONqYHdUem9Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=lists.ipfire.org; s=202003rsa; t=1785441119; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PEIWNxGDgetuya9TTchXxpiQoimJKS3j+M31Zuc5CkU=; b=o2evgCPigO9fkoobosB+TQO+WaAxQxZ7CwAYVKB98KwOmscEr4Z67sbR/m0gb0qFVb1hiK nRDce1pIKlfibDcDXpqqwcWlg5vSp7YNwCzbBX51B3JaRyoe3xVSu7IafRxMbxeeU5T+N4 m1rIWBNI4TgAt4q8zT9CTpU6c9XnOQYt5kvKkeNqBXvNOvAOAWJ8KgSlHq/P0nKt3+YtlT wDdzv7WIB99F1cJJb9heIg4C9DCSC20bHAcrbjsr9CM/jKZASQRLdXHnlUZPZCyxVCi63f FkH6EdD3NN86hBOfRRaktGtNxAcgQashYDO6gQ3At2fVeUJeCD87kBuPmdbpkw== ARC-Authentication-Results: i=1; mail01.ipfire.org; dkim=pass header.d=disroot.org header.s=mail header.b=M4wzHVJp; spf=pass (mail01.ipfire.org: domain of robin.roevens@disroot.org designates 178.21.23.139 as permitted sender) smtp.mailfrom=robin.roevens@disroot.org; dmarc=pass (policy=reject) header.from=disroot.org Received: from mail01.layka.lan (localhost [127.0.0.1]) by disroot.org (Postfix) with ESMTP id 487BF41B7F for ; Thu, 30 Jul 2026 21:51:59 +0200 (CEST) X-Virus-Scanned: SPAM Filter at disroot.org Received: from layka.disroot.org ([127.0.0.1]) by localhost (disroot.org [127.0.0.1]) (amavis, port 10024) with ESMTP id Ns3xmitg1yAW for ; Thu, 30 Jul 2026 21:51:58 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=disroot.org; s=mail; t=1785441118; bh=/kQ3ylxyioy6NKqB5KC32XhwINDSWBhHv79buPZlNPs=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=M4wzHVJpth+BoyBBM3tkDVDgiMSWABpZotRDbB5Tz9f3HgTCZAVPyCXpgwJz34lMB Ng0FvHF5gj7zF/xPyjwpiPAs0uVwHKn9vQMoOSHmhMC17belZudi98NBiaLDYb0d81 ++xycY+pgNtxoGzf/2j82ebQbBG9tqUjmG5eeUoYOjbCF/MX13ejJ+9Kucbb38d+U/ DozTjAab/jRBAgAvZow0pVcHAaE6GnZ5qj5hyqUb/KD6DHpMYQo71tVb8URP+ZMyKd ERLraorTCjkWJwG9aT1QT7Ex/RK2yuDO1xneY1rT0jgV/o2ZAxdbUJyydv6n6uo4EC H3HH327LIyxDA== Received: from chojin.roevenslambrechts.be (chojin.roevenslambrechts.be [192.168.0.50]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (no client certificate requested) (Authenticated sender) by hachiman (MailScanner Milter) with SMTP id 81984585FF8; Thu, 30 Jul 2026 21:51:55 +0200 (CEST) From: Robin Roevens To: development@lists.ipfire.org Cc: Robin Roevens Subject: [PATCH 4/5] Add function to send all pending alerts to Zabbix Date: Thu, 30 Jul 2026 21:15:55 +0200 Message-ID: <20260730195148.3278295-5-robin.roevens@disroot.org> In-Reply-To: <20260730195148.3278295-1-robin.roevens@disroot.org> References: <20260730195148.3278295-1-robin.roevens@disroot.org> Precedence: list List-Id: List-Subscribe: , List-Unsubscribe: , List-Post: List-Help: Sender: Mail-Followup-To: MIME-Version: 1.0 X-RoevensLambrechts-MailScanner-ID: 81984585FF8.AD543 X-RoevensLambrechts-MailScanner: Found to be clean X-RoevensLambrechts-MailScanner-From: robin.roevens@disroot.org X-RoevensLambrechts-MailScanner-Watermark: 1786045916.44132@YwP+JdYXgfW9L9UW9y3Nrw X-Rspamd-Server: mail01.haj.ipfire.org X-Rspamd-Queue-Id: 4hB0G74gfkz2h2 X-Rspamd-Action: no action X-Spamd-Result: default: False [-5.63 / 11.00]; BAYES_HAM(-3.00)[100.00%]; R_DKIM_ALLOW(-1.65)[disroot.org:s=mail]; MID_CONTAINS_FROM(1.00)[]; DKIM_REPUTATION(-0.92)[-0.92153870218341]; SPF_REPUTATION_HAM(-0.65)[-0.65402885146808]; DMARC_POLICY_ALLOW(-0.50)[disroot.org,reject]; R_MISSING_CHARSET(0.50)[]; R_SPF_ALLOW(-0.20)[+a:c]; MIME_GOOD(-0.10)[text/plain]; MX_GOOD(-0.10)[disroot.org]; RCPT_COUNT_TWO(0.00)[2]; ASN(0.00)[asn:50673, ipnet:178.21.23.0/24, country:NL]; IP_REPUTATION_HAM(0.00)[asn: 50673(0.00), country: NL(-0.01), ip: 178.21.23.139(0.00)]; ARC_NA(0.00)[]; TO_DN_SOME(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_THREE(0.00)[3]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MISSING_XM_UA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[disroot.org:+]; ARC_SIGNED(0.00)[lists.ipfire.org:s=202003rsa:i=1]; PREVIOUSLY_DELIVERED(0.00)[development@lists.ipfire.org]; FROM_HAS_DN(0.00)[] Add abbility to send all alerts from DB marked as pending to Zabbix server in bulk with errorhandling Signed-off-by: Robin Roevens --- src/suricata-reporter.in | 94 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 94 insertions(+) diff --git a/src/suricata-reporter.in b/src/suricata-reporter.in index 78bb04d..47482ab 100644 --- a/src/suricata-reporter.in +++ b/src/suricata-reporter.in @@ -326,6 +326,100 @@ class Reporter(object): # Commit it straight away self.db.commit() + async def flush_pending_to_zabbix(self): + """ + Sends all alerts that were marked for Zabbix in a single bulk call. + """ + if not self.zabbix_sender: + return False + + alert_item_hostname = self.config.get('zabbix', 'alert_item_hostname', fallback=(self.zabbix_sender.host or HOSTNAME)) + alert_item_key = self.config.get('zabbix', 'alert_item_key', fallback='ipfire.suricata.event.get') + + now = datetime.datetime.now() + alert_max_age = datetime.timedelta( + seconds = self.config.getint('zabbix', 'alert_max_age', fallback=3600) + ) + min_timestamp = (now - alert_max_age).timestamp() + + pending_rows = self.db.execute( + "SELECT id, event FROM alerts " + "WHERE zabbix_pending = 1 AND timestamp >= ? " + "ORDER BY id", + (min_timestamp,) + ).fetchall() + + if not pending_rows: + return True + + items = [] + item_ids = [] + + for alert_id, event_json in pending_rows: + try: + pending_event = Event(event_json) + except ValueError as e: + log.warning("Skipping malformed pending alert from database: %s" % e) + continue + + items.append(ItemValue( + alert_item_hostname, + alert_item_key, + pending_event.json, + int(pending_event.timestamp.timestamp()) + )) + item_ids.append(alert_id) + + if not items: + return True + + try: + # Send all pending items in bulk to Zabbix + response = await self.zabbix_sender.send(items) + + # Simulate zabbix_utils method of splitting the items into chunks + # so if a chunk fails to send, we know what items have failed + chunk_size = getattr(self.zabbix_sender, 'chunk_size', 250) + chunked_item_ids = [item_ids[i:i + chunk_size] for i in range(0, len(item_ids), chunk_size)] + successful_ids = [] + + # Check for failures, determine which chunks where sent successfully + if response.failed == 0: + successful_ids = item_ids + elif response.details: + for node, chunks in response.details.items(): + for chunk_index, resp in enumerate(chunks): + chunk_ids = chunked_item_ids[chunk_index] + if resp.failed == 0: + log.debug(f"Zabbix sender: Pending chunk sent successfully to {node} in {resp.time}") + successful_ids.extend(chunk_ids) + else: + log.error(f"Zabbix sender: Failed to send pending chunk to {node} at chunk {resp.chunk}") + log.debug(response) + else: + log.error(f"Zabbix sender: Failed to send {len(item_ids)} pending alerts.") + log.debug(response) + + # Mark sent items as no longer pending in the DB + if successful_ids: + self.db.execute( + "UPDATE alerts SET zabbix_pending = 0 WHERE id IN ({})".format( + ", ".join("?" for _ in successful_ids) + ), + successful_ids + ) + self.db.commit() + + log.debug(f"Zabbix sender: {len(successful_ids)} alerts sent successfully") + pending_count = len(item_ids) - len(successful_ids) + if pending_count != 0: + log.debug(f"Zabbix sender: {pending_count} alerts failed to send and are still pending") + return pending_count == 0 + + except Exception as e: + log.error(f"Zabbix sender: Failed to send {len(item_ids)} pending alerts: {e}") + return False + def optimize(self): """ Called when the process exits to optimize the database From patchwork Thu Jul 30 19:15:56 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robin Roevens X-Patchwork-Id: 10081 Return-Path: Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (Client CN "mail01.haj.ipfire.org", Issuer "YR2" (not verified)) by web04.haj.ipfire.org (Postfix) with ESMTPS id 4hB0Lw4ggzz3wqJ for ; Thu, 30 Jul 2026 19:56:08 +0000 (UTC) Received: from mail02.haj.ipfire.org (mail02.haj.ipfire.org [172.28.1.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (Client CN "mail02.haj.ipfire.org", Issuer "YE1" (not verified)) by mail01.ipfire.org (Postfix) with ESMTPS id 4hB0Lm357Yz5gj for ; Thu, 30 Jul 2026 19:56:00 +0000 (UTC) Received: from mail02.haj.ipfire.org (localhost [IPv6:::1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4hB0HK6llGz36WB for ; Thu, 30 Jul 2026 19:53:01 +0000 (UTC) X-Original-To: development@lists.ipfire.org Received: from mail01.ipfire.org (mail01.haj.ipfire.org [IPv6:2001:678:b28::25]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (Client CN "mail01.haj.ipfire.org", Issuer "YR2" (not verified)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4hB0GS3XwVz37B0 for ; Thu, 30 Jul 2026 19:52:16 +0000 (UTC) Received: from layka.disroot.org (layka.disroot.org [178.21.23.139]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (Client did not present a certificate) by mail01.ipfire.org (Postfix) with ESMTPS id 4hB0GH0T3bz2V for ; Thu, 30 Jul 2026 19:52:07 +0000 (UTC) Authentication-Results: mail01.ipfire.org; dkim=pass header.d=disroot.org header.s=mail header.b="C/BhnjPn"; spf=pass (mail01.ipfire.org: domain of robin.roevens@disroot.org designates 178.21.23.139 as permitted sender) smtp.mailfrom=robin.roevens@disroot.org; dmarc=pass (policy=reject) header.from=disroot.org ARC-Seal: i=1; a=rsa-sha256; d=lists.ipfire.org; s=202003rsa; cv=none; t=1785441127; b=dHIm5+hjNrPSBu+lczlrN1nQXY94hqb+VGBLODJydNMfl9wRUd+6Bbg7aagsTYUjxtu9Fr q8A06b7hRruThH3gspEES5FX7Qn/PLrvKgJD5Y/QlKWVE61aAR2i6pP+wQ+mE8SnhaMcmV 7fm14b+e7OY0m6jbGuwewZaENHT8aRQ1YzB1TrKs8Fs7xWL+oBKTVSWOha7Q8qB848USpP byhnJ+86gdBkAqiRHGs6ga3mCQ4Cm6kleIVv+NxZtNPrvZdwTA24NsMsqyaekgR0kGJoDV GQkrz3WaOAQGjrLT3E53PoDTsqhgQScALtQPRMc7R7BBQ1qhm4bqifLgu9Xxsw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=lists.ipfire.org; s=202003rsa; t=1785441127; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Lvv+RJk/W5YzW3xktjsJsIOwHRtJ3GhQum9nGBke6cQ=; b=IUmaM/SgYarFAsO11WD7CFsrngwAShaVuxz4h8CcTcIxRRS7HOz75m3Lhy2RhoO3+6fOBo cOlBtxjV/gEuf8H9gvA56R/VY3jN4hmHC6Ez9DAb86uwCuTZtn8iIq4wCBCerWuQBXc6pJ ekoIKsvk8wzh4lkBBqQVtosF5Qgyiq02QCufNlgLsAgipljPK7+lYNLyuKmYrd1Wm/AYUl I4ls54lF1060rJZ+M7icGcDsD1r//5JzwWXf5ltyEctT9228ryl1Lqr/LQeQzCAuPXe5bw lb6MIgigH5ihgA3T9tn8eEMaa6IAzBBecppeKif2nqWysUCbmD9FPrgQJZgxyg== ARC-Authentication-Results: i=1; mail01.ipfire.org; dkim=pass header.d=disroot.org header.s=mail header.b="C/BhnjPn"; spf=pass (mail01.ipfire.org: domain of robin.roevens@disroot.org designates 178.21.23.139 as permitted sender) smtp.mailfrom=robin.roevens@disroot.org; dmarc=pass (policy=reject) header.from=disroot.org Received: from mail01.layka.lan (localhost [127.0.0.1]) by disroot.org (Postfix) with ESMTP id 2414A84BE3 for ; Thu, 30 Jul 2026 21:52:01 +0200 (CEST) X-Virus-Scanned: SPAM Filter at disroot.org Received: from layka.disroot.org ([127.0.0.1]) by localhost (disroot.org [127.0.0.1]) (amavis, port 10024) with ESMTP id rMU8qh7r5RZc for ; Thu, 30 Jul 2026 21:51:59 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=disroot.org; s=mail; t=1785441119; bh=NURcr4XnAFuUDfdeapqS8IpXOdi9JFq/WhhmERhP0DY=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=C/BhnjPnX5VpjT8UddvYFbDAkEiM4XJ8dhksu4veadcz8E2pEdTsRmq6vFTPMHT+l 6yWOYyG6Gp7vecD6Qu0giuMjOtzz+uWXglNVxjMlQWlzw2/gxIYOlTePItkb5RjwU6 fwbVLa/Om1vDFYSZGNzOD1xZR7xWaCxIUUd5XmN/hgu0rycGWDXhkgg05Ao7SlTrrS nDjt83ySjsn3B6252LwuJv0EOUuUbgOawEtPZVKtE+1KAdxd266vtmPtEg+ihfr70Q 9Fo2nfIMq9bcnILoIDarmVhj4fmg6TNcfSWU6h+J2+6aG7ZhfMArdMqdxbzb1PQLQR XDw+mdluAW3Qg== Received: from chojin.roevenslambrechts.be (chojin.roevenslambrechts.be [192.168.0.50]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (no client certificate requested) (Authenticated sender) by hachiman (MailScanner Milter) with SMTP id 50C23586005; Thu, 30 Jul 2026 21:51:56 +0200 (CEST) From: Robin Roevens To: development@lists.ipfire.org Cc: Robin Roevens Subject: [PATCH 5/5] Add background task to send all pending alerts to Zabbix every 1 second Date: Thu, 30 Jul 2026 21:15:56 +0200 Message-ID: <20260730195148.3278295-6-robin.roevens@disroot.org> In-Reply-To: <20260730195148.3278295-1-robin.roevens@disroot.org> References: <20260730195148.3278295-1-robin.roevens@disroot.org> Precedence: list List-Id: List-Subscribe: , List-Unsubscribe: , List-Post: List-Help: Sender: Mail-Followup-To: MIME-Version: 1.0 X-RoevensLambrechts-MailScanner-ID: 50C23586005.AD543 X-RoevensLambrechts-MailScanner: Found to be clean X-RoevensLambrechts-MailScanner-From: robin.roevens@disroot.org X-RoevensLambrechts-MailScanner-Watermark: 1786045917.0423@2NZt8INRh+ADTxeDe/V5PA X-Rspamd-Server: mail01.haj.ipfire.org X-Rspamd-Queue-Id: 4hB0GH0T3bz2V X-Rspamd-Action: no action X-Spamd-Result: default: False [-5.63 / 11.00]; BAYES_HAM(-2.99)[99.97%]; R_DKIM_ALLOW(-1.65)[disroot.org:s=mail]; MID_CONTAINS_FROM(1.00)[]; DKIM_REPUTATION(-0.92)[-0.92153661380942]; SPF_REPUTATION_HAM(-0.66)[-0.65685405590104]; DMARC_POLICY_ALLOW(-0.50)[disroot.org,reject]; R_MISSING_CHARSET(0.50)[]; R_SPF_ALLOW(-0.20)[+a:c]; MIME_GOOD(-0.10)[text/plain]; MX_GOOD(-0.10)[disroot.org]; ASN(0.00)[asn:50673, ipnet:178.21.23.0/24, country:NL]; ARC_SIGNED(0.00)[lists.ipfire.org:s=202003rsa:i=1]; TO_DN_SOME(0.00)[]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; MISSING_XM_UA(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_COUNT_THREE(0.00)[3]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; IP_REPUTATION_HAM(0.00)[asn: 50673(0.00), country: NL(-0.01), ip: 178.21.23.139(0.00)]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[development@lists.ipfire.org]; DKIM_TRACE(0.00)[disroot.org:+] Background task that will send pending events every 1 second. On failure it will retry 3 times and then pauses until next alert comes in to prevent hammering a possibly already overloaded Zabbix server.. Signed-off-by: Robin Roevens --- src/suricata-reporter.in | 109 +++++++++++++++++++++++++++++++++------ 1 file changed, 94 insertions(+), 15 deletions(-) diff --git a/src/suricata-reporter.in b/src/suricata-reporter.in index 47482ab..a95f6b7 100644 --- a/src/suricata-reporter.in +++ b/src/suricata-reporter.in @@ -82,10 +82,6 @@ class Reporter(object): # Remember the last time the database was cleaned self.last_cleanup_at = None - # Initialize Zabbix sender - self.zabbix_sender = None - self.init_zabbix_sender() - # Register any signals for signo in (signal.SIGINT, signal.SIGTERM): self.loop.add_signal_handler(signo, self.terminate) @@ -99,6 +95,11 @@ class Reporter(object): # Create the socket self.sock = self._create_socket() + # Initialize Zabbix sender + self.zabbix_sender = None + self.init_zabbix_sender() + self.zabbix_sender_task = None + def read_config(self): """ Reads or re-reads the configuration. @@ -123,6 +124,12 @@ class Reporter(object): zabbix_server_host = self.config.get('zabbix', 'zabbix_server_host', fallback='') zabbix_server_port = self.config.getint('zabbix', 'zabbix_server_port', fallback=10051) + # Zabbix sender backoff state + self.zabbix_consecutive_failures = 0 + self.zabbix_pause_until_new_event = False + self.zabbix_sender_wakeup = asyncio.Event() + self.zabbix_pause_after_failures = 3 + if zabbix_config: if not os.path.isfile(zabbix_config): log.error(f"Zabbix agent config file {zabbix_config} does not exist.") @@ -242,6 +249,67 @@ class Reporter(object): # Return the socket return sock + def _start_zabbix_sender_task(self): + """ + Start a background Zabbix sender task + """ + if not self.config.getboolean("zabbix", "enabled", fallback=False): + return + + self.zabbix_sender_task = asyncio.create_task(self._periodic_zabbix_sender_flush()) + + async def _stop_zabbix_sender_task(self): + """ + Stops the background Zabbix sender task + """ + if not self.zabbix_sender_task: + return + + self.zabbix_sender_task.cancel() + try: + await self.zabbix_sender_task + except asyncio.CancelledError: + pass + + # Final flush of any remaining alerts + await self.flush_pending_to_zabbix() + + async def _periodic_zabbix_sender_flush(self): + """ + Background task that periodically sends all pending Zabbix alerts. + Runs every 1 second to batch-send alerts in bulk on heavy load, but + pauses after repeated failures until a new event arrives. + """ + log.debug("Starting periodic Zabbix sender task") + + try: + while not self.is_terminated.is_set(): + if self.zabbix_pause_until_new_event: + log.warning("Zabbix sending is paused after repeated failures; waiting for the next event") + self.zabbix_sender_wakeup.clear() + await self.zabbix_sender_wakeup.wait() + self.zabbix_sender_wakeup.clear() + self.zabbix_pause_until_new_event = False + self.zabbix_consecutive_failures = 0 + continue + + if self.is_terminated.is_set(): + break + + # Send pending alerts to Zabbix + if await self.flush_pending_to_zabbix(): + self.zabbix_consecutive_failures = 0 + else: + self.zabbix_consecutive_failures += 1 + if self.zabbix_consecutive_failures >= self.zabbix_pause_after_failures: + self.zabbix_pause_until_new_event = True + + await asyncio.sleep(1) + + except asyncio.CancelledError: + log.debug("Periodic Zabbix sender task cancelled") + raise + async def run(self): """ The main loop of the application. @@ -251,22 +319,29 @@ class Reporter(object): # Cleanup the database at startup self.cleanup() - # Wait until we have terminated - await self.is_terminated.wait() + # Start the periodic Zabbix sender task + self._start_zabbix_sender_task() - # Remove the socket so we won't receive any more data try: - os.unlink(self.socket_path) - except OSError as e: - log.error("Failed to remove %s: %s" % (self.socket_path, e)) + # Wait until we have terminated + await self.is_terminated.wait() + finally: + # Remove the socket so we won't receive any more data + try: + os.unlink(self.socket_path) + except OSError as e: + log.error("Failed to remove %s: %s" % (self.socket_path, e)) + + # Cancel the periodic Zabbix sender task + await self._stop_zabbix_sender_task() - # We will optimize the database before we exit - self.optimize() + # We will optimize the database before we exit + self.optimize() - # Close the database - self.db.close() + # Close the database + self.db.close() - log.debug("Reporter has exited") + log.debug("Reporter has exited") def terminate(self): """ @@ -485,6 +560,10 @@ class Reporter(object): # Store the alert self.store(event) + # Wake the periodic flush task so it can retry immediately on new input + if self.config.getboolean("zabbix", "enabled", fallback=False): + self.zabbix_sender_wakeup.set() + # Send to syslog if self.config.getboolean("syslog", "enabled", fallback=False): await self.send_to_syslog(event)