From patchwork Sat Dec 26 17:09:23 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Schantl X-Patchwork-Id: 3762 Return-Path: Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature RSA-PSS (4096 bits)) (Client CN "mail01.haj.ipfire.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by web04.haj.ipfire.org (Postfix) with ESMTPS id 4D39KJ2mMpz3wxc for ; Sat, 26 Dec 2020 17:09:40 +0000 (UTC) Received: from mail02.haj.ipfire.org (mail02.haj.ipfire.org [172.28.1.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail02.haj.ipfire.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mail01.ipfire.org (Postfix) with ESMTPS id 4D39K86NsQz3bb; Sat, 26 Dec 2020 17:09:32 +0000 (UTC) Received: from mail02.haj.ipfire.org (localhost [127.0.0.1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4D39K825P8z2yPl; Sat, 26 Dec 2020 17:09:32 +0000 (UTC) Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature RSA-PSS (4096 bits)) (Client CN "mail01.haj.ipfire.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4D39K71VpWz2xYh for ; Sat, 26 Dec 2020 17:09:31 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by mail01.ipfire.org (Postfix) with ESMTPSA id 4D39K63LWsz8F; Sat, 26 Dec 2020 17:09:30 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003ed25519; t=1609002570; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=b/OxgHyX6RQEb+dsQEtz7Jq2eQ13w+9aZbEe7qgDziU=; b=GV7y+dAWEWy9dd+hFH7RxXgfzUNKjZtOlbT5KWxMk8pmWsC1mvXxXZxDlgAzJmJJjD5G/y wIrU3fgNa9/JXNBg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003rsa; t=1609002570; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=b/OxgHyX6RQEb+dsQEtz7Jq2eQ13w+9aZbEe7qgDziU=; b=g5B71vSLgR5jCeor6tkj+qsTHdzrH8M4/AoDbtHjn51x2stBSeuCdFGrJ9i3SYCjLNa1qF Ew82qMNNGZ6Hazxu3AS6bDI2YYsE+IxFpjPaSF2eDBwGuF6zzxPYK2GR9FtXxTRcXX+4+m 0TFk6H0RNYUZx9NJfM95N87FiqGMDQDAhGpktGl5YZ/j0RS4Rt+GJ+FSrotED9IUFaFycZ u/B/HUCAyIsFSa0zy9FkgNtL8seGPdLiVZSoFuqseprMvZLqFlrDW6k3Hp+FiyO62kucHj dPYif3SnPuQuX9QX9XkiCwntNOY3HIxIOcCnjkhSzh6MuFlCbV91vAjwIkNHTA== From: Stefan Schantl To: development@lists.ipfire.org Subject: [PATCH 1/2] ninja: Allow to limit the parallel build processes. Date: Sat, 26 Dec 2020 18:09:23 +0100 Message-Id: <20201226170924.12208-1-stefan.schantl@ipfire.org> MIME-Version: 1.0 X-BeenThere: development@lists.ipfire.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: IPFire development talk List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: development-bounces@lists.ipfire.org Sender: "Development" When run, ninja normally runs a maximum number of processes in parallel. By default this is the number of cores on the system plus two. In some cases this can overheat a CPU or run a system out of memory. If run from the command line, passing a -jN parameter will limit the number of parallel processes, but some packages embed the execution of ninja and do not pass a -j parameter. Using this optional procedure allows us to limit the number of parallel processes via an environment variable, NINJAJOBS. Signed-off-by: Stefan Schantl --- lfs/ninja | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/lfs/ninja b/lfs/ninja index c968345f4..ed9f8b778 100644 --- a/lfs/ninja +++ b/lfs/ninja @@ -71,6 +71,14 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects)) @$(PREBUILD) @rm -rf $(DIR_APP) && cd $(DIR_SRC) && tar zxf $(DIR_DL)/$(DL_FILE) + # Allow to limit the paralell build processes ninja will do. + cd $(DIR_APP) && sed -i '/int Guess/a \ + int j = 0;\ + char* jobs = getenv( "NINJAJOBS" );\ + if ( jobs != NULL ) j = atoi( jobs );\ + if ( j > 0 ) return j;\ + ' src/ninja.cc + # Configure and bootstrap the build environment cd $(DIR_APP) && python3 configure.py --bootstrap --verbose From patchwork Sat Dec 26 17:09:24 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Schantl X-Patchwork-Id: 3761 Return-Path: Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature RSA-PSS (4096 bits)) (Client CN "mail01.haj.ipfire.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by web04.haj.ipfire.org (Postfix) with ESMTPS id 4D39KJ2W3cz3wxY for ; Sat, 26 Dec 2020 17:09:40 +0000 (UTC) Received: from mail02.haj.ipfire.org (mail02.haj.ipfire.org [172.28.1.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail02.haj.ipfire.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mail01.ipfire.org (Postfix) with ESMTPS id 4D39K931sfz4QD; Sat, 26 Dec 2020 17:09:33 +0000 (UTC) Received: from mail02.haj.ipfire.org (localhost [127.0.0.1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4D39K92lj7z2xm2; Sat, 26 Dec 2020 17:09:33 +0000 (UTC) Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature RSA-PSS (4096 bits)) (Client CN "mail01.haj.ipfire.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4D39K810cBz2xkD for ; Sat, 26 Dec 2020 17:09:32 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by mail01.ipfire.org (Postfix) with ESMTPSA id 4D39K73D1Yz1Vm; Sat, 26 Dec 2020 17:09:31 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003ed25519; t=1609002571; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=94NUNfIYhATNXTY+cInigLIW98E2AEsU/YVx5m9KzRs=; b=oi0bwFFkNPxl0vBmSLVzUlsNKVTiRG9g8CE9rRUWHMkK53umvmm/UsHUkJw44eLhwBjN0z 8G6lqXQKnvcG8vAw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003rsa; t=1609002571; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=94NUNfIYhATNXTY+cInigLIW98E2AEsU/YVx5m9KzRs=; b=ad5/71+qwfdx5zEr9TcHOQ7ZfOBNi0dDY8HzulPwuMavMRdn1dG28/og7hhiviocp+ZKOy RFxSPFQnmZJMjYvUS22L/P9RwnX0ZI2ZsjDEKxlTo0rBgFPwG5mIWUIHfmeqO8iV5xf1b+ 2fgLup2tNdEWT3YZ7eegqwkAFYFSu1yZdYgTrIrhbkX8dktKdniRmJ6ZyYWkYImB/BB0gr 0Y+6BL2+JuNmmJefG6mZQyQnk2HU/N7nNOYMtxyOtsY7rPSGu4i9RibK5TeqJ2UskHuIQa wwbvqrOqYdj4LRJnkC1d5IaXb0CBPxCdmuURQYxjCsdIX8Iw5pRZ3PCzi+L3Tw== From: Stefan Schantl To: development@lists.ipfire.org Subject: [PATCH 2/2] make.sh: Limit ninja to compute paralellism value Date: Sat, 26 Dec 2020 18:09:24 +0100 Message-Id: <20201226170924.12208-2-stefan.schantl@ipfire.org> In-Reply-To: <20201226170924.12208-1-stefan.schantl@ipfire.org> References: <20201226170924.12208-1-stefan.schantl@ipfire.org> MIME-Version: 1.0 X-BeenThere: development@lists.ipfire.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: IPFire development talk List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: development-bounces@lists.ipfire.org Sender: "Development" Signed-off-by: Stefan Schantl --- make.sh | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/make.sh b/make.sh index 5b1e949c4..8dfc8897d 100755 --- a/make.sh +++ b/make.sh @@ -220,6 +220,9 @@ configure_build() { # Use this as default PARALLELISM DEFAULT_PARALLELISM="${parallelism}" + # Limit lauched ninja build jobs to computed parallel value. + NINJAJOBS="${parallelism}" + # Compression parameters # We use mode 8 for reasonable memory usage when decompressing # but with overall good compression @@ -465,7 +468,7 @@ prepareenv() { # Setup environment set +h LC_ALL=POSIX - export LFS LC_ALL CFLAGS CXXFLAGS DEFAULT_PARALLELISM RUSTFLAGS + export LFS LC_ALL CFLAGS CXXFLAGS DEFAULT_PARALLELISM RUSTFLAGS NINJAJOBS unset CC CXX CPP LD_LIBRARY_PATH LD_PRELOAD # Make some extra directories