From patchwork Sun May 20 17:22:59 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthias Fischer X-Patchwork-Id: 1777 Return-Path: Received: from mail01.ipfire.org (unknown [172.28.1.200]) by web02.i.ipfire.org (Postfix) with ESMTP id 1593A60A28 for ; Sun, 20 May 2018 09:23:11 +0200 (CEST) Received: from mail01.i.ipfire.org (localhost [IPv6:::1]) by mail01.ipfire.org (Postfix) with ESMTP id 7A26810D22D4; Sun, 20 May 2018 08:23:10 +0100 (BST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfire.org; s=201801; t=1526800990; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:references:list-id: list-unsubscribe:list-subscribe:list-post; bh=ys/HhM2cDg2/FZSx0Ry/ignha6qzIC7D6Q1ai9cGU8I=; b=FQjw9ZWw+aQSAioWXwXBkKLaFfj27rGyDAV999wsme6ysaYntZg1LatPRp7AP6sv8Y9bhm sSuAxjxOUZeh2FGcRvYlGQlqZ0yXSxxoc9ouB7skXn1uyjj0ElgaZdqQ2Ryzx7SCVTflst NVc9QHpQkNrjNNY89o4ehq2Uu2A4BbhjEALxSTsHTU00nO76wtUAHvlE0npIkrh1/jmAML /ns0NgzhRf8ImTIDIk92v0awmT+0vRLA1UICcTANB61Itn9BpUwzQFO/HYXdE0wudGxBUn aMJx0Ynlfmr7v0Ghk1JzsF/0rFroHer19xLwQScJfw0n3x1/4qqY3nnJhvBC1g== Authentication-Results: mail01.i.ipfire.org; auth=pass smtp.auth=mfischer smtp.mailfrom=matthias.fischer@ipfire.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfire.org; s=201801; t=1526800985; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:references; bh=ys/HhM2cDg2/FZSx0Ry/ignha6qzIC7D6Q1ai9cGU8I=; b=o1MFvGbG+DV9UNIHFQ3Ea6m3QZoYppnkPZimxl1mtqodKQ2qpytaK/i+FmPHWO7PGXNfdM W5eTn62CN2qVz8Ii6CpHcL1vbpgrJrI9P/pDdAfqBvEnD5WDVoOYw1hE8fvkLV6HTfOtJe 19sqljUobz5a93FpKCdRTBtmrCG9cV8sYiYNDyHHBB8beQcikoUPqk9Pb1f8ZOkKkABZUS AyYlm4JYjVm76DwqTThJ2KVJkwPipYpjM8wsb4VGSBy09WyglwLXtQ78noVIqyw98/QOrv GpQTWrGdN7yOw5H3RYycQiEd3YDn+QRMqJuJQZ1h9gTzdUjoG8hg2TCR5f75qA== Received: from Devel.localdomain (p5B0A2E81.dip0.t-ipconnect.de [91.10.46.129]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client did not present a certificate) by mail01.ipfire.org (Postfix) with ESMTPSA id C78A610D22CB for ; Sun, 20 May 2018 08:23:04 +0100 (BST) From: Matthias Fischer To: development@lists.ipfire.org Subject: [PATCH] xz: Performance and compression tuning (V1) Date: Sun, 20 May 2018 09:22:59 +0200 Message-Id: <20180520072259.5253-1-matthias.fischer@ipfire.org> X-Mailer: git-send-email 2.17.0 X-Spamd-Result: default: False [-5.02 / 11.00]; ASN(0.00)[asn:3320, ipnet:91.0.0.0/10, country:DE]; DKIM_SIGNED(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_TLS_ALL(0.00)[]; BAYES_HAM(-3.00)[100.00%]; ARC_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MID_CONTAINS_FROM(1.00)[]; RCVD_COUNT_ZERO(0.00)[0]; RCPT_COUNT_ONE(0.00)[1]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; NEURAL_HAM(-2.92)[-0.972,0] X-Spam-Status: No, score=-5.02 X-Rspamd-Server: mail01.i.ipfire.org X-BeenThere: development@lists.ipfire.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: IPFire development talk List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: development-bounces@lists.ipfire.org Sender: "Development" Hi, as mentioned in the "list", we're trying to make the archives as small as possible using 'xz'-compression. In order to achieve this, this patch tests the size of the memory available on the host system. It sets the xz-memory-limit (--memory=[N]Mib) to 70% of the available working memory, a maximum of four xz-'worker threads' (-T4) and a compression rate of '-8' (-8). These options are written to the 'XZ_OPT=' environment. For details see: https://linux.die.net/man/1/xz I have set the number of available xz-'worker threads' (-T) to four (-T4), because during the final tests the '-T0' parameter led to error messages snd stopped: 'cannot allocate memory'. It wouldn't even run with 90%. Furthermore, testing with '-T0' led to countless messages filling up '_build.packages.log'. E.g.: ... xz: Adjusted the number of threads from 8 to 2 to not exceed the memory usage limit of 1557 MiB ... Tests took place on a 32bit-Ubuntu 16.04.4-system with 8 GB RAM and an Intel I7-2600. Build time was about 04:30 hrs. Perhaps a 64bit-system would perform better (higher values), but my goal was to make this run on as many systems as possible, so I choosed these averages. If minimum requirements (1024 MB RAM) are not met, building stops. Current results: 'next', untuned: ipfire-2.19.2gb-ext4.i586-full-core121.img.gz => 332951687 Bytes ipfire-2.19.i586-full-core121.iso => 228589568 Bytes 'next', XZ_OPT: -T4 -8, 70% RAM: ipfire-2.19.2gb-ext4.i586-full-core121.img.gz => 329725723 Bytes ipfire-2.19.i586-full-core121.iso => 217055232 Bytes These two resulting archives are 14760300 Bytes smaller than before. Best, Matthias Signed-off-by: Matthias Fischer --- lfs/Config | 2 +- lfs/cdrom | 5 +---- make.sh | 18 +++++++++++++++++- 3 files changed, 19 insertions(+), 6 deletions(-) diff --git a/lfs/Config b/lfs/Config index d63a1dc9a..c667a3cc7 100644 --- a/lfs/Config +++ b/lfs/Config @@ -209,7 +209,7 @@ define PAK tar xf /install/packages/package/files.tmp -C /install/packages/package/tmp/ \ -p --numeric-owner rm -f /install/packages/package/files.tmp - cd /install/packages/package/tmp/ && XZ_OPT=-T0 tar -c -p --numeric-owner -J -f /install/packages/package/files.tar.xz * + cd /install/packages/package/tmp/ && XZ_OPT="$(XZ_OPT)" tar -c -p --numeric-owner -J -f /install/packages/package/files.tar.xz * rm -r /install/packages/package/tmp -cat /install/packages/package/ROOTFILES | grep -v "#" > /install/packages/package/ROOTFILES.tmp mv /install/packages/package/ROOTFILES.tmp /install/packages/package/ROOTFILES diff --git a/lfs/cdrom b/lfs/cdrom index 7a7fff166..820c55b87 100644 --- a/lfs/cdrom +++ b/lfs/cdrom @@ -35,9 +35,6 @@ else TAR_OPTIONS = --xz endif -# Enable multi-threaded compression for LZMA -export XZ_OPT = --threads=0 - ############################################################################### # Top-level Rules ############################################################################### @@ -74,7 +71,7 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects)) # Compress root filesystem # Reason for this tar+untar+tar is removing of entries listed two or more in src/ROOTFILES - tar -c -C / --files-from=/tmp/ROOTFILES \ + XZ_OPT="$(XZ_OPT)" tar -c -C / --files-from=/tmp/ROOTFILES \ -f /$(SNAME).tar --exclude='#*' --exclude='dev/pts/*' \ --exclude='proc/*' --exclude='tmp/ROOTFILES' rm -f /tmp/ROOTFILES diff --git a/make.sh b/make.sh index 419f9fa95..b6f3f37d2 100755 --- a/make.sh +++ b/make.sh @@ -526,6 +526,7 @@ enterchroot() { CCACHE_COMPRESS="${CCACHE_COMPRESS}" \ CCACHE_COMPILERCHECK="${CCACHE_COMPILERCHECK}" \ KVER="${KVER}" \ + XZ_OPT="${XZ_OPT}" \ $(fake_environ) \ $(qemu_environ) \ "$@" @@ -860,6 +861,21 @@ fi # Get the amount of memory in this build system HOST_MEM=$(system_memory) +# Checking host memory, tuning XZ_OPT + +if [ $HOST_MEM -lt 16384 ]; then + print_build_stage "Host-Memory: $HOST_MEM MiB" + print_build_stage "Not enough host memory (less than 1024 MiB, please consider upgrading)" + + exit 1 + +else + + XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB" + XZ_OPT="-T4 -8 --memory=$XZ_MEM" + +fi + if [ -n "${BUILD_ARCH}" ]; then configure_build "${BUILD_ARCH}" else @@ -1760,7 +1776,7 @@ toolchain) buildtoolchain echo "`date -u '+%b %e %T'`: Create toolchain image for ${BUILD_ARCH}" | tee -a $LOGFILE test -d $BASEDIR/cache/toolchains || mkdir -p $BASEDIR/cache/toolchains - cd $BASEDIR && XZ_OPT="-T0 -8" tar -Jc --exclude='log/_build.*.log' -f cache/toolchains/$SNAME-$VERSION-toolchain-$TOOLCHAINVER-${BUILD_ARCH}.tar.xz \ + cd $BASEDIR && XZ_OPT="$(XZ_OPT)" tar -Jc --exclude='log/_build.*.log' -f cache/toolchains/$SNAME-$VERSION-toolchain-$TOOLCHAINVER-${BUILD_ARCH}.tar.xz \ build/${TOOLS_DIR} build/bin/sh log >> $LOGFILE md5sum cache/toolchains/$SNAME-$VERSION-toolchain-$TOOLCHAINVER-${BUILD_ARCH}.tar.xz \ > cache/toolchains/$SNAME-$VERSION-toolchain-$TOOLCHAINVER-${BUILD_ARCH}.md5