From patchwork Thu Jul 30 19:15:55 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robin Roevens X-Patchwork-Id: 10078 Return-Path: Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (Client CN "mail01.haj.ipfire.org", Issuer "YR2" (not verified)) by web04.haj.ipfire.org (Postfix) with ESMTPS id 4hB0J50TdWz3wqJ for ; Thu, 30 Jul 2026 19:53:41 +0000 (UTC) Received: from mail02.haj.ipfire.org (mail02.haj.ipfire.org [172.28.1.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (Client CN "mail02.haj.ipfire.org", Issuer "YE1" (not verified)) by mail01.ipfire.org (Postfix) with ESMTPS id 4hB0Hw6Gn7z6G0 for ; Thu, 30 Jul 2026 19:53:32 +0000 (UTC) Received: from mail02.haj.ipfire.org (localhost [IPv6:::1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4hB0GL2P6sz3771 for ; Thu, 30 Jul 2026 19:52:10 +0000 (UTC) X-Original-To: development@lists.ipfire.org Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (Client CN "mail01.haj.ipfire.org", Issuer "YR2" (not verified)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4hB0GH5dHHz33bJ for ; Thu, 30 Jul 2026 19:52:07 +0000 (UTC) Received: from layka.disroot.org (layka.disroot.org [178.21.23.139]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (Client did not present a certificate) by mail01.ipfire.org (Postfix) with ESMTPS id 4hB0G74gfkz2h2 for ; Thu, 30 Jul 2026 19:51:59 +0000 (UTC) Authentication-Results: mail01.ipfire.org; dkim=pass header.d=disroot.org header.s=mail header.b=M4wzHVJp; spf=pass (mail01.ipfire.org: domain of robin.roevens@disroot.org designates 178.21.23.139 as permitted sender) smtp.mailfrom=robin.roevens@disroot.org; dmarc=pass (policy=reject) header.from=disroot.org ARC-Seal: i=1; a=rsa-sha256; d=lists.ipfire.org; s=202003rsa; cv=none; t=1785441119; b=mmVfG3Iqh77NmSppT6lqVzwZMM/woRFbLdyb7UtnsVTnRcodZ45iXCVQGwMFMmAKEoJiqL M0LtcwkdPHLGvo38eb2jdw7IeDhScNzlcH1HlhpCCS9DMJmW7F1niaKviL60wRdobu2Ir9 VHfM3Cp/kgRMZTpyBIJ9WtapDM3kDT5FfaiOQu57/Sk/sclgKMyYFGJ9o4nklTPpx+oLkx cZJwnq7JtwsrcbOZDdc+RC8ijQGAVQEpqlAfYmymEz6G5LE0M/zHAOOK5+VrRKnPkBWnHg +15B2DqtkdfWl0mApre+F4ZZmFJLgfEGsNiG7w8QxeZ77Nk12SjONqYHdUem9Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=lists.ipfire.org; s=202003rsa; t=1785441119; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PEIWNxGDgetuya9TTchXxpiQoimJKS3j+M31Zuc5CkU=; b=o2evgCPigO9fkoobosB+TQO+WaAxQxZ7CwAYVKB98KwOmscEr4Z67sbR/m0gb0qFVb1hiK nRDce1pIKlfibDcDXpqqwcWlg5vSp7YNwCzbBX51B3JaRyoe3xVSu7IafRxMbxeeU5T+N4 m1rIWBNI4TgAt4q8zT9CTpU6c9XnOQYt5kvKkeNqBXvNOvAOAWJ8KgSlHq/P0nKt3+YtlT wDdzv7WIB99F1cJJb9heIg4C9DCSC20bHAcrbjsr9CM/jKZASQRLdXHnlUZPZCyxVCi63f FkH6EdD3NN86hBOfRRaktGtNxAcgQashYDO6gQ3At2fVeUJeCD87kBuPmdbpkw== ARC-Authentication-Results: i=1; mail01.ipfire.org; dkim=pass header.d=disroot.org header.s=mail header.b=M4wzHVJp; spf=pass (mail01.ipfire.org: domain of robin.roevens@disroot.org designates 178.21.23.139 as permitted sender) smtp.mailfrom=robin.roevens@disroot.org; dmarc=pass (policy=reject) header.from=disroot.org Received: from mail01.layka.lan (localhost [127.0.0.1]) by disroot.org (Postfix) with ESMTP id 487BF41B7F for ; Thu, 30 Jul 2026 21:51:59 +0200 (CEST) X-Virus-Scanned: SPAM Filter at disroot.org Received: from layka.disroot.org ([127.0.0.1]) by localhost (disroot.org [127.0.0.1]) (amavis, port 10024) with ESMTP id Ns3xmitg1yAW for ; Thu, 30 Jul 2026 21:51:58 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=disroot.org; s=mail; t=1785441118; bh=/kQ3ylxyioy6NKqB5KC32XhwINDSWBhHv79buPZlNPs=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=M4wzHVJpth+BoyBBM3tkDVDgiMSWABpZotRDbB5Tz9f3HgTCZAVPyCXpgwJz34lMB Ng0FvHF5gj7zF/xPyjwpiPAs0uVwHKn9vQMoOSHmhMC17belZudi98NBiaLDYb0d81 ++xycY+pgNtxoGzf/2j82ebQbBG9tqUjmG5eeUoYOjbCF/MX13ejJ+9Kucbb38d+U/ DozTjAab/jRBAgAvZow0pVcHAaE6GnZ5qj5hyqUb/KD6DHpMYQo71tVb8URP+ZMyKd ERLraorTCjkWJwG9aT1QT7Ex/RK2yuDO1xneY1rT0jgV/o2ZAxdbUJyydv6n6uo4EC H3HH327LIyxDA== Received: from chojin.roevenslambrechts.be (chojin.roevenslambrechts.be [192.168.0.50]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (no client certificate requested) (Authenticated sender) by hachiman (MailScanner Milter) with SMTP id 81984585FF8; Thu, 30 Jul 2026 21:51:55 +0200 (CEST) From: Robin Roevens To: development@lists.ipfire.org Cc: Robin Roevens Subject: [PATCH 4/5] Add function to send all pending alerts to Zabbix Date: Thu, 30 Jul 2026 21:15:55 +0200 Message-ID: <20260730195148.3278295-5-robin.roevens@disroot.org> In-Reply-To: <20260730195148.3278295-1-robin.roevens@disroot.org> References: <20260730195148.3278295-1-robin.roevens@disroot.org> Precedence: list List-Id: List-Subscribe: , List-Unsubscribe: , List-Post: List-Help: Sender: Mail-Followup-To: MIME-Version: 1.0 X-RoevensLambrechts-MailScanner-ID: 81984585FF8.AD543 X-RoevensLambrechts-MailScanner: Found to be clean X-RoevensLambrechts-MailScanner-From: robin.roevens@disroot.org X-RoevensLambrechts-MailScanner-Watermark: 1786045916.44132@YwP+JdYXgfW9L9UW9y3Nrw X-Rspamd-Server: mail01.haj.ipfire.org X-Rspamd-Queue-Id: 4hB0G74gfkz2h2 X-Rspamd-Action: no action X-Spamd-Result: default: False [-5.63 / 11.00]; BAYES_HAM(-3.00)[100.00%]; R_DKIM_ALLOW(-1.65)[disroot.org:s=mail]; MID_CONTAINS_FROM(1.00)[]; DKIM_REPUTATION(-0.92)[-0.92153870218341]; SPF_REPUTATION_HAM(-0.65)[-0.65402885146808]; DMARC_POLICY_ALLOW(-0.50)[disroot.org,reject]; R_MISSING_CHARSET(0.50)[]; R_SPF_ALLOW(-0.20)[+a:c]; MIME_GOOD(-0.10)[text/plain]; MX_GOOD(-0.10)[disroot.org]; RCPT_COUNT_TWO(0.00)[2]; ASN(0.00)[asn:50673, ipnet:178.21.23.0/24, country:NL]; IP_REPUTATION_HAM(0.00)[asn: 50673(0.00), country: NL(-0.01), ip: 178.21.23.139(0.00)]; ARC_NA(0.00)[]; TO_DN_SOME(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_THREE(0.00)[3]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MISSING_XM_UA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[disroot.org:+]; ARC_SIGNED(0.00)[lists.ipfire.org:s=202003rsa:i=1]; PREVIOUSLY_DELIVERED(0.00)[development@lists.ipfire.org]; FROM_HAS_DN(0.00)[] Add abbility to send all alerts from DB marked as pending to Zabbix server in bulk with errorhandling Signed-off-by: Robin Roevens --- src/suricata-reporter.in | 94 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 94 insertions(+) diff --git a/src/suricata-reporter.in b/src/suricata-reporter.in index 78bb04d..47482ab 100644 --- a/src/suricata-reporter.in +++ b/src/suricata-reporter.in @@ -326,6 +326,100 @@ class Reporter(object): # Commit it straight away self.db.commit() + async def flush_pending_to_zabbix(self): + """ + Sends all alerts that were marked for Zabbix in a single bulk call. + """ + if not self.zabbix_sender: + return False + + alert_item_hostname = self.config.get('zabbix', 'alert_item_hostname', fallback=(self.zabbix_sender.host or HOSTNAME)) + alert_item_key = self.config.get('zabbix', 'alert_item_key', fallback='ipfire.suricata.event.get') + + now = datetime.datetime.now() + alert_max_age = datetime.timedelta( + seconds = self.config.getint('zabbix', 'alert_max_age', fallback=3600) + ) + min_timestamp = (now - alert_max_age).timestamp() + + pending_rows = self.db.execute( + "SELECT id, event FROM alerts " + "WHERE zabbix_pending = 1 AND timestamp >= ? " + "ORDER BY id", + (min_timestamp,) + ).fetchall() + + if not pending_rows: + return True + + items = [] + item_ids = [] + + for alert_id, event_json in pending_rows: + try: + pending_event = Event(event_json) + except ValueError as e: + log.warning("Skipping malformed pending alert from database: %s" % e) + continue + + items.append(ItemValue( + alert_item_hostname, + alert_item_key, + pending_event.json, + int(pending_event.timestamp.timestamp()) + )) + item_ids.append(alert_id) + + if not items: + return True + + try: + # Send all pending items in bulk to Zabbix + response = await self.zabbix_sender.send(items) + + # Simulate zabbix_utils method of splitting the items into chunks + # so if a chunk fails to send, we know what items have failed + chunk_size = getattr(self.zabbix_sender, 'chunk_size', 250) + chunked_item_ids = [item_ids[i:i + chunk_size] for i in range(0, len(item_ids), chunk_size)] + successful_ids = [] + + # Check for failures, determine which chunks where sent successfully + if response.failed == 0: + successful_ids = item_ids + elif response.details: + for node, chunks in response.details.items(): + for chunk_index, resp in enumerate(chunks): + chunk_ids = chunked_item_ids[chunk_index] + if resp.failed == 0: + log.debug(f"Zabbix sender: Pending chunk sent successfully to {node} in {resp.time}") + successful_ids.extend(chunk_ids) + else: + log.error(f"Zabbix sender: Failed to send pending chunk to {node} at chunk {resp.chunk}") + log.debug(response) + else: + log.error(f"Zabbix sender: Failed to send {len(item_ids)} pending alerts.") + log.debug(response) + + # Mark sent items as no longer pending in the DB + if successful_ids: + self.db.execute( + "UPDATE alerts SET zabbix_pending = 0 WHERE id IN ({})".format( + ", ".join("?" for _ in successful_ids) + ), + successful_ids + ) + self.db.commit() + + log.debug(f"Zabbix sender: {len(successful_ids)} alerts sent successfully") + pending_count = len(item_ids) - len(successful_ids) + if pending_count != 0: + log.debug(f"Zabbix sender: {pending_count} alerts failed to send and are still pending") + return pending_count == 0 + + except Exception as e: + log.error(f"Zabbix sender: Failed to send {len(item_ids)} pending alerts: {e}") + return False + def optimize(self): """ Called when the process exits to optimize the database