From patchwork Tue May 22 00:19:48 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthias Fischer X-Patchwork-Id: 1781 Return-Path: Received: from mail01.ipfire.org (mail01.ipfire.org [IPv6:2001:470:7183:25::1]) by web02.i.ipfire.org (Postfix) with ESMTP id AB4766099D for ; Mon, 21 May 2018 16:19:56 +0200 (CEST) Received: from mail01.i.ipfire.org (localhost [IPv6:::1]) by mail01.ipfire.org (Postfix) with ESMTP id 18F34110ACCB; Mon, 21 May 2018 15:19:56 +0100 (BST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfire.org; s=201801; t=1526912396; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:references:list-id: list-unsubscribe:list-subscribe:list-post; bh=42j3l9mCiN8/9A6lgeC1+/PoPlmo8ZyT5/RMJ7PdEDM=; b=CpGPCHO7iy28T7tkIFTZSXIbGbzfqOaOYSw1js2x4wSuLLB32dyGLrhmiKQyBB7cH8iur9 m12A+QrGe9a4xJCLeOEjdm/RpYfP275xbgB78J4v+4yWjynt9+1htLexLn0vspcEUjVu/P z2RJmQ3TDDDC7/mZH83n35BYHanu1si8hPZ+petmrvfePba8f+qGq/zjwLFvD2v3KftKbh fDRHMG03/T/t+n7Sj9mXY6VjIpSg4bOF/u6OBs5T7kyoBDrcrvaUvEF3mLMFaOmXW3XliB vk6+IoLBZ0CQLpGLlUREXzlUtaepDptshfV0FXfhYVkjCnisXyizMeSVI5RsTw== Authentication-Results: mail01.i.ipfire.org; auth=pass smtp.auth=mfischer smtp.mailfrom=matthias.fischer@ipfire.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfire.org; s=201801; t=1526912393; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:references; bh=42j3l9mCiN8/9A6lgeC1+/PoPlmo8ZyT5/RMJ7PdEDM=; b=Q12T9ri2YfVofZmoYLn84XG6znsuJzRw3dP/GiGJfnfJoDL1eb2rlmzbIGxkY3DyERNQTJ hDyRBczCsjBIi/frf/AQJnAFimMU47sNd3ceWi5KhJHHOqyESgpKl15dAX1LmOrjOclB40 MP2K1WBFX3+nWQ7EGhsSRyHLJH8XGRUY9+HWIKQqiwZ1N2PWlcE1JWtTOniWbkyFGIOhaK x71Dm3HeuROJOZs+CjpVl/7Iy2A5CP4wUSyBRUpuOu/RX9mPdCzgqholbpXO9f+ngQ2cq2 dOiIo0hz/F0mcevFDrTQz4wv7IOfCWcwxiEaWZO6Guw3p1vegUWgYaaiqZ17Dw== Received: from Devel.localdomain (p5B0A2E81.dip0.t-ipconnect.de [91.10.46.129]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client did not present a certificate) by mail01.ipfire.org (Postfix) with ESMTPSA id 14FBD10D22C5 for ; Mon, 21 May 2018 15:19:52 +0100 (BST) From: Matthias Fischer To: development@lists.ipfire.org Subject: [PATCH] xz: Performance and compression tuning (V2) - fine tuning Date: Mon, 21 May 2018 16:19:48 +0200 Message-Id: <20180521141948.10024-1-matthias.fischer@ipfire.org> X-Mailer: git-send-email 2.17.0 X-Spamd-Result: default: False [-5.02 / 11.00]; ASN(0.00)[asn:3320, ipnet:91.0.0.0/10, country:DE]; DKIM_SIGNED(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_TLS_ALL(0.00)[]; BAYES_HAM(-3.00)[100.00%]; ARC_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MID_CONTAINS_FROM(1.00)[]; RCVD_COUNT_ZERO(0.00)[0]; RCPT_COUNT_ONE(0.00)[1]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; NEURAL_HAM(-2.92)[-0.972,0] X-Spam-Status: No, score=-5.02 X-Rspamd-Server: mail01.i.ipfire.org X-BeenThere: development@lists.ipfire.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: IPFire development talk List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: development-bounces@lists.ipfire.org Sender: "Development" Changes since V1: - Tuned some more 'tar'-command lines - Included 'lfs/core-updates' - Some fine-tuning Hi, Current results with V2 (clean builds): 'next': 'packages' => 255 MB 'ipfire-2.19.2gb-ext4.i586-full-core121.img.gz => 319 MB 'ipfire-2.19.i586-full-core121.iso' => 218 MB Total => 792 MB 'xz-tuning': 'packages' => 228 MB 'ipfire-2.19.2gb-ext4.i586-full-core121.img.gz' => 318 MB 'ipfire-2.19.i586-full-core121.iso' => 207 MB Total => 753 MB (-39 MB) It would be nice if someone could review and test these. ;-) If necessary, I can upload a complete patch. Best, Matthias Signed-off-by: Matthias Fischer --- lfs/Config | 6 ++++-- lfs/cdrom | 4 ++-- lfs/core-updates | 9 +++++---- make.sh | 2 +- 4 files changed, 12 insertions(+), 9 deletions(-) diff --git a/lfs/Config b/lfs/Config index c667a3cc7..a726f171d 100644 --- a/lfs/Config +++ b/lfs/Config @@ -47,6 +47,8 @@ else PREFIX = /usr endif +TAR_OPTIONS = --xz + # URLs that are common sources of downloads. If you're having trouble with # a site you should change its URL to that of a suitable mirror site. # @@ -209,11 +211,11 @@ define PAK tar xf /install/packages/package/files.tmp -C /install/packages/package/tmp/ \ -p --numeric-owner rm -f /install/packages/package/files.tmp - cd /install/packages/package/tmp/ && XZ_OPT="$(XZ_OPT)" tar -c -p --numeric-owner -J -f /install/packages/package/files.tar.xz * + cd /install/packages/package/tmp/ && XZ_OPT="$(XZ_OPT)" tar -c -p --numeric-owner -J -f /install/packages/package/files.tar.xz $(TAR_OPTIONS) * rm -r /install/packages/package/tmp -cat /install/packages/package/ROOTFILES | grep -v "#" > /install/packages/package/ROOTFILES.tmp mv /install/packages/package/ROOTFILES.tmp /install/packages/package/ROOTFILES - cd /install/packages/package && tar cf ../$(PROG)-$(VER)-$(PAK_VER).ipfire --files-from=/usr/src/src/paks/files + cd /install/packages/package && XZ_OPT="$(XZ_OPT)" tar cf ../$(PROG)-$(VER)-$(PAK_VER).ipfire $(TAR_OPTIONS) --files-from=/usr/src/src/paks/files rm -rf /install/packages/package sed -e s/NAME/$(PROG)/g \ -e s/VER/$(VER)/g \ diff --git a/lfs/cdrom b/lfs/cdrom index 820c55b87..cdb7b07da 100644 --- a/lfs/cdrom +++ b/lfs/cdrom @@ -71,14 +71,14 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects)) # Compress root filesystem # Reason for this tar+untar+tar is removing of entries listed two or more in src/ROOTFILES - XZ_OPT="$(XZ_OPT)" tar -c -C / --files-from=/tmp/ROOTFILES \ + tar -c -C / --files-from=/tmp/ROOTFILES \ -f /$(SNAME).tar --exclude='#*' --exclude='dev/pts/*' \ --exclude='proc/*' --exclude='tmp/ROOTFILES' rm -f /tmp/ROOTFILES tar -x -C /tmp -f /$(SNAME).tar rm -f /$(SNAME).tar @mkdir /tmp/sys - cd /tmp && tar cf /install/cdrom/distro.img $(TAR_OPTIONS) * && rm -rf * + cd /tmp && XZ_OPT="$(XZ_OPT)" tar cf /install/cdrom/distro.img $(TAR_OPTIONS) * && rm -rf * # Other files touch /install/cdrom/$(SNAME)-$(VERSION)-core$(CORE).media diff --git a/lfs/core-updates b/lfs/core-updates index 3c847daf2..d34926967 100644 --- a/lfs/core-updates +++ b/lfs/core-updates @@ -26,8 +26,9 @@ include Config VER = ipfire -THISAPP = core-updates -TARGET = $(DIR_INFO)/$(THISAPP) +THISAPP = core-updates +TARGET = $(DIR_INFO)/$(THISAPP) +TAR_OPTIONS = --xz ############################################################################### # Top-level Rules @@ -78,7 +79,7 @@ core/%: mv -f /tmp/ROOTFILES /install/packages/package/ROOTFILES tar -p --numeric-owner -x -C /tmp -f /$(SNAME).tar rm -f /$(SNAME).tar - cd /tmp && tar cJf /install/packages/package/files.tar.xz * && rm -rf * + cd /tmp && XZ_OPT="$(XZ_OPT)" tar cJf /install/packages/package/files.tar.xz $(TAR_OPTIONS) * && rm -rf * cat /install/packages/package/ROOTFILES | grep -v "#" > /install/packages/package/ROOTFILES.tmp rm /install/packages/package/ROOTFILES mv /install/packages/package/ROOTFILES.tmp /install/packages/package/ROOTFILES @@ -89,7 +90,7 @@ core/%: #Change xxxKVERxxx to Kernelversion sed -e "s/xxxKVERxxx/$(KVER)/g" -i /install/packages/package/update.sh - cd /install/packages/package && tar cf ../core-upgrade-$(VERSION)-$$(basename $@).ipfire \ + cd /install/packages/package && XZ_OPT="$(XZ_OPT)" tar cf ../core-upgrade-$(VERSION)-$$(basename $@).ipfire $(TAR_OPTIONS) \ update.sh files.tar.xz ROOTFILES rm -rf /install/packages/package sed -e "s/NAME/core-upgrade/g" \ diff --git a/make.sh b/make.sh index 33ebc16bb..9893fbb20 100755 --- a/make.sh +++ b/make.sh @@ -1776,7 +1776,7 @@ toolchain) buildtoolchain echo "`date -u '+%b %e %T'`: Create toolchain image for ${BUILD_ARCH}" | tee -a $LOGFILE test -d $BASEDIR/cache/toolchains || mkdir -p $BASEDIR/cache/toolchains - cd $BASEDIR && XZ_OPT="$(XZ_OPT)" tar -Jc --exclude='log/_build.*.log' -f cache/toolchains/$SNAME-$VERSION-toolchain-$TOOLCHAINVER-${BUILD_ARCH}.tar.xz \ + cd $BASEDIR && XZ_OPT="$(XZ_OPT)" tar -Jc --exclude='log/_build.*.log' -f cache/toolchains/$SNAME-$VERSION-toolchain-$TOOLCHAINVER-${BUILD_ARCH}.tar.xz --xz \ build/${TOOLS_DIR} build/bin/sh log >> $LOGFILE md5sum cache/toolchains/$SNAME-$VERSION-toolchain-$TOOLCHAINVER-${BUILD_ARCH}.tar.xz \ > cache/toolchains/$SNAME-$VERSION-toolchain-$TOOLCHAINVER-${BUILD_ARCH}.md5