build: xz tarball extreme compression #10626

PeterDaveHello · 2017-01-05T05:45:33Z

Take node v7.4.0 as example, tarball size improvements listed as below:

node-v7.4.0-darwin-x64.tar.xz     9176904 ->  9147884 (99.68%)
node-v7.4.0-headers.tar.xz         351224 ->   349612 (99.54%)
node-v7.4.0-linux-arm64.tar.xz    9271000 ->  9254748 (99.82%)
node-v7.4.0-linux-armv6l.tar.xz   9243504 ->  9227428 (99.82%)
node-v7.4.0-linux-armv7l.tar.xz   9246228 ->  9228732 (99.81%)
node-v7.4.0-linux-ppc64.tar.xz    9448476 ->  9411128 (99.60%)
node-v7.4.0-linux-ppc64le.tar.xz  9553876 ->  9521424 (99.66%)
node-v7.4.0-linux-s390x.tar.xz    9923212 ->  9901772 (99.78%)
node-v7.4.0-linux-x64.tar.xz     10318700 -> 10304396 (99.86%)
node-v7.4.0-linux-x86.tar.xz      9907848 ->  9886448 (99.78%)
node-v7.4.0-sunos-x86.tar.xz      9742620 ->  9732160 (99.89%)
node-v7.4.0.tar.xz               16611356 -> 16459192 (99.08%)

So that we can know that we can have the improvement on all the xz
tarball releases!

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
commit message follows commit guidelines

Affected core subsystem(s)

build

mscdex · 2017-01-05T06:22:36Z

According to the xz manpage, using -e can increase the compression time "dramatically (it can easily double)."

Is that really worth it for <=1% difference?

PeterDaveHello · 2017-01-05T06:27:01Z

@mscdex I think it depends the download counts on the server, personally I don't feel the time increase so much, take node-v7.4.0-linux-x64.tar as example on my CPU E3-1220 V2 CPU:

$ time xz -9 node-v7.4.0-linux-x64.tar 

real    0m20.632s
user    0m20.432s
sys     0m0.192s

$ time xz -9e node-v7.4.0-linux-x64.tar                                   

real    0m27.529s
user    0m27.378s
sys     0m0.124s

The time increases 35% more, but still only few (7) seconds 😄

gibfahn · 2017-01-05T08:20:35Z

So the average size decrease is 0.31%? I'm not sure how useful that'd be.

PeterDaveHello · 2017-01-05T08:26:02Z

It's not that significant improvement, but that 7 secs(on my computer, may be even shorter on nodejs's build server) could save more than 7 secs in total around the world if there are enough downloads count, and I guess it'll 😄

Since the tarball is static, once it's released, it'll not be touched anymore in the most cases, and could be download by millions of times (depends on the time window), even if it can only save ~ 30KB, 30KB x 1000000 = ~ 3GB, so we can every 3GB bandwidth from every million times download, from a large scale of view, the ~ 7 secs may worth it?

gibfahn · 2017-01-05T09:59:47Z

@PeterDaveHello So it's 7 seconds more on your computer to compress? How much longer does it take to extract?

PeterDaveHello · 2017-01-05T21:54:45Z

@gibfahn I didn't see significant difference from the decompression time, 1.287 secs vs 1.281 secs.

jasnell · 2017-03-24T23:15:38Z

Updates on this one?

PeterDaveHello · 2017-03-25T08:53:43Z

Same question :)

bnoordhuis · 2017-03-26T09:54:45Z

Decompression time should be the same if I understand xz's 'extreme' algorithm correctly, but the resulting file could be either bigger or smaller than without -e.

It would be interesting to see if our tarballs are persistently smaller across releases or if it's hit and miss. Any volunteers?

PeterDaveHello · 2017-03-26T11:26:51Z

@bnoordhuis across all released versions and different architecture?

bnoordhuis · 2017-03-26T11:33:56Z

All releases might be a bit excessive but I'd be curious to see the numbers for a few releases from the v4.x, v6.x and v7.x release branches each.

PeterDaveHello · 2017-03-26T11:51:51Z

v7 already been tested as above, I'll test v4 & v6 later

PeterDaveHello · 2017-03-26T12:28:29Z

node-v4.8.1-darwin-x64.tar.xz            7124752 ->  7149424 (99.6549%)
node-v4.8.1-headers.tar.xz                338352 ->   339964 (99.5258%)
node-v4.8.1-linux-arm64.tar.xz           7550036 ->  7560260 (99.8648%)
node-v4.8.1-linux-armv6l.tar.xz          7455440 ->  7470656 (99.7963%)
node-v4.8.1-linux-armv7l.tar.xz          7458396 ->  7468584 (99.8636%)
node-v4.8.1-linux-ppc64le.tar.xz         7720644 ->  7742768 (99.7143%)
node-v4.8.1-linux-ppc64.tar.xz           7596784 ->  7624740 (99.6334%)
node-v4.8.1-linux-x64.tar.xz             8307500 ->  8318712 (99.8652%)
node-v4.8.1-linux-x86.tar.xz             7932348 ->  7944156 (99.8514%)
node-v4.8.1-sunos-x64.tar.xz             8492692 ->  8520216 (99.6777%)
node-v4.8.1-sunos-x86.tar.xz             7854796 ->  7890112 (99.5524%)
node-v4.8.1.tar.xz                      13155440 -> 13293124 (98.9642%)

PeterDaveHello · 2017-03-26T12:35:59Z

node-v6.10.1-darwin-x64.tar.xz     8320216 ->  8290400 (99.6416%)
node-v6.10.1-headers.tar.xz         347024 ->   345344 (99.5159%)
node-v6.10.1-linux-arm64.tar.xz    8432120 ->  8422480 (99.8857%)
node-v6.10.1-linux-armv6l.tar.xz   8312748 ->  8287768 (99.6995%)
node-v6.10.1-linux-armv7l.tar.xz   8320636 ->  8295836 (99.7019%)
node-v6.10.1-linux-ppc64le.tar.xz  8674404 ->  8646288 (99.6759%)
node-v6.10.1-linux-ppc64.tar.xz    8536804 ->  8506192 (99.6414%)
node-v6.10.1-linux-s390x.tar.xz    8955096 ->  8924860 (99.6624%)
node-v6.10.1-linux-x64.tar.xz      9361912 ->  9347648 (99.8476%)
node-v6.10.1-linux-x86.tar.xz      8956748 ->  8941476 (99.8295%)
node-v6.10.1-sunos-x64.tar.xz      9540948 ->  9519380 (99.7739%)
node-v6.10.1-sunos-x86.tar.xz      8828132 ->  8844696 (1.00188%)
node-v6.10.1.tar.xz               15746356 -> 15590420 (99.0097%)

bnoordhuis · 2017-03-27T08:54:22Z

Thanks, @PeterDaveHello. Okay, so it's a win most of the time. Let's enable it but can you add it to the XZ_COMPRESSION variable on (or around) line 600 instead of passing it manually everywhere?

Take node v7.4.0 as example, tarball size improvements listed as below: node-v7.4.0-darwin-x64.tar.xz 9176904 -> 9147884 (99.68%) node-v7.4.0-headers.tar.xz 351224 -> 349612 (99.54%) node-v7.4.0-linux-arm64.tar.xz 9271000 -> 9254748 (99.82%) node-v7.4.0-linux-armv6l.tar.xz 9243504 -> 9227428 (99.82%) node-v7.4.0-linux-armv7l.tar.xz 9246228 -> 9228732 (99.81%) node-v7.4.0-linux-ppc64.tar.xz 9448476 -> 9411128 (99.60%) node-v7.4.0-linux-ppc64le.tar.xz 9553876 -> 9521424 (99.66%) node-v7.4.0-linux-s390x.tar.xz 9923212 -> 9901772 (99.78%) node-v7.4.0-linux-x64.tar.xz 10318700 -> 10304396 (99.86%) node-v7.4.0-linux-x86.tar.xz 9907848 -> 9886448 (99.78%) node-v7.4.0-sunos-x86.tar.xz 9742620 -> 9732160 (99.89%) node-v7.4.0.tar.xz 16611356 -> 16459192 (99.08%) So that we can know that we can have the improvement on all the xz tarball releases!

PeterDaveHello · 2017-03-27T09:06:25Z

@bnoordhuis done.

bnoordhuis

Thanks, LGTM. Probably no point in running the CI, I don't think it tests tarball creation.

fhinkel · 2017-05-26T19:07:58Z

Note to self (or any other collaborator): This is ready to merge, removing the stalled label.

PeterDaveHello · 2017-05-26T19:34:35Z

Thanks @fhinkel

PR-URL: nodejs#10626 Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Franziska Hinkelmann <[email protected]>

refack · 2017-05-27T21:30:51Z

Landed in 1474b7a

PR-URL: #10626 Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Franziska Hinkelmann <[email protected]>

MylesBorins · 2017-07-17T17:08:49Z

landed on v6.x, let me know if it should be backed out

PR-URL: #10626 Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Franziska Hinkelmann <[email protected]>

nodejs-github-bot added build Issues and PRs related to build files or the CI. lts-watch-v6.x labels Jan 5, 2017

jasnell added the stalled Issues and PRs that are stalled. label Mar 24, 2017

nodejs-github-bot removed lts-watch-v6.x labels Mar 27, 2017

bnoordhuis approved these changes Mar 27, 2017

View reviewed changes

refack force-pushed the master branch from 16073c0 to fbe946b Compare April 14, 2017 04:11

fhinkel approved these changes May 26, 2017

View reviewed changes

fhinkel removed the stalled Issues and PRs that are stalled. label May 26, 2017

refack pushed a commit to refack/node that referenced this pull request May 27, 2017

build: xz tarball extreme compression

1474b7a

PR-URL: nodejs#10626 Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Franziska Hinkelmann <[email protected]>

refack closed this May 27, 2017

PeterDaveHello deleted the improve-xz-compression branch May 27, 2017 22:33

jasnell pushed a commit that referenced this pull request May 28, 2017

build: xz tarball extreme compression

fb6cf2f

PR-URL: #10626 Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Franziska Hinkelmann <[email protected]>

jasnell mentioned this pull request May 28, 2017

8.0.0 Release Proposal #12220

Closed

gibfahn mentioned this pull request Jun 15, 2017

Auditing for 6.11.1 nodejs/Release#230

Closed

3 tasks

MylesBorins added the land-on-v6.x label Jul 17, 2017

MylesBorins pushed a commit that referenced this pull request Jul 17, 2017

build: xz tarball extreme compression

ebbde61

PR-URL: #10626 Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Franziska Hinkelmann <[email protected]>

MylesBorins mentioned this pull request Jul 18, 2017

v6.11.2 proposal #14356

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

build: xz tarball extreme compression #10626

build: xz tarball extreme compression #10626

PeterDaveHello commented Jan 5, 2017 •

edited

Loading

mscdex commented Jan 5, 2017

PeterDaveHello commented Jan 5, 2017 •

edited

Loading

gibfahn commented Jan 5, 2017

PeterDaveHello commented Jan 5, 2017 •

edited

Loading

gibfahn commented Jan 5, 2017

PeterDaveHello commented Jan 5, 2017

jasnell commented Mar 24, 2017

PeterDaveHello commented Mar 25, 2017

bnoordhuis commented Mar 26, 2017

PeterDaveHello commented Mar 26, 2017

bnoordhuis commented Mar 26, 2017

PeterDaveHello commented Mar 26, 2017 •

edited

Loading

PeterDaveHello commented Mar 26, 2017

PeterDaveHello commented Mar 26, 2017

bnoordhuis commented Mar 27, 2017

PeterDaveHello commented Mar 27, 2017

bnoordhuis left a comment

fhinkel commented May 26, 2017

PeterDaveHello commented May 26, 2017

refack commented May 27, 2017

MylesBorins commented Jul 17, 2017

build: xz tarball extreme compression #10626

build: xz tarball extreme compression #10626

Conversation

PeterDaveHello commented Jan 5, 2017 • edited Loading

Checklist

Affected core subsystem(s)

mscdex commented Jan 5, 2017

PeterDaveHello commented Jan 5, 2017 • edited Loading

gibfahn commented Jan 5, 2017

PeterDaveHello commented Jan 5, 2017 • edited Loading

gibfahn commented Jan 5, 2017

PeterDaveHello commented Jan 5, 2017

jasnell commented Mar 24, 2017

PeterDaveHello commented Mar 25, 2017

bnoordhuis commented Mar 26, 2017

PeterDaveHello commented Mar 26, 2017

bnoordhuis commented Mar 26, 2017

PeterDaveHello commented Mar 26, 2017 • edited Loading

PeterDaveHello commented Mar 26, 2017

PeterDaveHello commented Mar 26, 2017

bnoordhuis commented Mar 27, 2017

PeterDaveHello commented Mar 27, 2017

bnoordhuis left a comment

Choose a reason for hiding this comment

fhinkel commented May 26, 2017

PeterDaveHello commented May 26, 2017

refack commented May 27, 2017

MylesBorins commented Jul 17, 2017

PeterDaveHello commented Jan 5, 2017 •

edited

Loading

PeterDaveHello commented Jan 5, 2017 •

edited

Loading

PeterDaveHello commented Jan 5, 2017 •

edited

Loading

PeterDaveHello commented Mar 26, 2017 •

edited

Loading