aboutsummaryrefslogtreecommitdiff
path: root/src/compress
AgeCommit message (Collapse)Author
2024-04-26compress: reordering fields to reduce struct sizesapocelipes
Overall, there are 32 bytes reduced. Change-Id: I455bf0874b33fa47719f42618e4800c7ff2a9e88 GitHub-Last-Rev: 7670344c4a643afdec0fdae3d34fdb8ccd81205f GitHub-Pull-Request: golang/go#67010 Reviewed-on: https://go-review.googlesource.com/c/go/+/581355 Reviewed-by: Joedian Reid <joedian@google.com> Reviewed-by: Keith Randall <khr@google.com> Auto-Submit: Keith Randall <khr@golang.org> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Keith Randall <khr@golang.org>
2024-04-15compress/bzip2: simplify Huffman tree constructionaimuz
This change simplifies the construction of the Huffman tree in the bzip2 package by replacing custom sort logic with the more concise and idiomatic use of "slices" and "cmp" packages. Change-Id: I2a8aef146b54b9433038b133d2cc8856ba077c72 GitHub-Last-Rev: c031bb56635552205c12b87add5f4337a7def74f GitHub-Pull-Request: golang/go#66817 Reviewed-on: https://go-review.googlesource.com/c/go/+/578438 Reviewed-by: Ian Lance Taylor <iant@google.com> Auto-Submit: Ian Lance Taylor <iant@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Cherry Mui <cherryyz@google.com>
2024-04-10crypto/internal/mlkem768: new packageFilippo Valsorda
This was initially developed at github.com/FiloSottile/mlkem768. + 5ce9162 - mlkem768,xwing: add SeedSize <Filippo Valsorda> + b43add9 - mlkem768,xwing: add NewKeyFromSeed <Filippo Valsorda> + e000fa4 - mlkem768: improve RoundTrip benchmark <Filippo Valsorda> + 344d5ee - mlkem768: add exhaustive tests for compress and decompress (#4) <David Buchanan> + 08fb36c - mlkem768: do not panic <Filippo Valsorda> + 9e9fcc2 - mlkem768: add proposed Wycheproof test vectors <Filippo Valsorda> + 5e630b8 - mlkem768: add more tests <Filippo Valsorda> + e3fb5df - mlkem768: add TestUnluckyVector <Filippo Valsorda> + 3f410e9 - mlkem768: add accumulated pq-crystals vectors <Filippo Valsorda> + 9897e2f - mlkem768: add other known test vectors <Filippo Valsorda> + cffbfb9 - mlkem768: update sampleNTT comment <Filippo Valsorda> + df1b265 - mlkem768: use uint16 reads, simpler bit twiddling <Josh Bleecher Snyder> + 50a7fad - mlkem768: unroll ntt inner loop <Josh Bleecher Snyder> + cd8140e - mlkem768: avoid extra data copies <Josh Bleecher Snyder> + 0c68443 - mlkem768: buffer reads from sha3 <Josh Bleecher Snyder> + bb784ff - mlkem768: create README.md <Filippo Valsorda> + 35e7ada - mlkem768: add package docs and LICENSE <Filippo Valsorda> + 2e6a3df - mlkem768: drop performance optimization notes <Filippo Valsorda> + d5449de - mlkem768: add benchmarks <Filippo Valsorda> + 3294fee - mlkem768: implement ML-KEM <Filippo Valsorda> + 4cb306e - mlkem768: reimplement compress and decompress <Filippo Valsorda> + 48e4c4c - mlkem768: fix AHat draft spec typo <Filippo Valsorda> + c34ddcf - mlkem768: make better use of constants <Filippo Valsorda> + 3b485e1 - mlkem768: initial commit, a full K-PKE implementation <Filippo Valsorda> Submitting changes on behalf of Josh Bleecher Snyder as authorized at https://go-review.googlesource.com/c/go/+/547357/comment/61f8433f_04dc9c5d/ and of David Buchanan as authorized at https://github.com/FiloSottile/mlkem768/pull/4#issuecomment-1975330952. Updates #64537 Change-Id: I50607336282434d64a1255901b0ef40dbfd47e91 Reviewed-on: https://go-review.googlesource.com/c/go/+/550215 Reviewed-by: Roland Shoemaker <roland@golang.org> Reviewed-by: Damien Neil <dneil@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Auto-Submit: Filippo Valsorda <filippo@golang.org>
2024-03-18compress/bzip2,lzw: use built-in clear to simplify codeapocelipes
Change-Id: I16c17e322c757c8c657364065948d7cec66a8346 GitHub-Last-Rev: 9a5104fe9874dd7c604c526be3f082487f2aaf01 GitHub-Pull-Request: golang/go#66377 Reviewed-on: https://go-review.googlesource.com/c/go/+/572199 Reviewed-by: Keith Randall <khr@google.com> Reviewed-by: David Chase <drchase@google.com> Reviewed-by: Joseph Tsai <joetsai@digital-static.net> Auto-Submit: Keith Randall <khr@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: qiulaidongfeng <2645477756@qq.com> Reviewed-by: Jorropo <jorropo.pgm@gmail.com> Reviewed-by: Keith Randall <khr@golang.org>
2024-03-11compress/gzip: close writer in testguoguangwu
Change-Id: I12bc9287106f1492cbc9e74b4163cce97c957d31 GitHub-Last-Rev: cda1b48fe3ee9083a2262f1d6eeb039c66c12b40 GitHub-Pull-Request: golang/go#66185 Reviewed-on: https://go-review.googlesource.com/c/go/+/569896 Reviewed-by: Cherry Mui <cherryyz@google.com> Reviewed-by: Ian Lance Taylor <iant@google.com> Reviewed-by: Joseph Tsai <joetsai@digital-static.net> Auto-Submit: Ian Lance Taylor <iant@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2023-10-13compress: some minor corrections or adjustments for godoc linkcui fliter
Change-Id: Ic1fc777b9a4e8262851e27646d53a0ec4fe55b64 Reviewed-on: https://go-review.googlesource.com/c/go/+/535076 Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Ian Lance Taylor <iant@google.com> Run-TryBot: shuang cui <imcusg@gmail.com> Auto-Submit: Ian Lance Taylor <iant@google.com> Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Joseph Tsai <joetsai@digital-static.net> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2023-10-12compress: add available godoc linkcui fliter
Change-Id: Ia6e88aec59cb294e8b303a00fcd69f4cbf0dc09a Reviewed-on: https://go-review.googlesource.com/c/go/+/534759 Reviewed-by: Ian Lance Taylor <iant@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Auto-Submit: Ian Lance Taylor <iant@google.com> Run-TryBot: shuang cui <imcusg@gmail.com>
2023-09-01compress: linkify references to io.ByteReaderJoe Tsai
The special treatment of io.ByteReader is somewhat significant. Linkify references to that type from documentation. Change-Id: I77c06e4c6b957d107638e2ff2a4c58148b396ed3 Reviewed-on: https://go-review.googlesource.com/c/go/+/525016 Reviewed-by: Matthew Dempsky <mdempsky@google.com> Run-TryBot: Joseph Tsai <joetsai@digital-static.net> Reviewed-by: Ian Lance Taylor <iant@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Auto-Submit: Joseph Tsai <joetsai@digital-static.net>
2023-06-16compress/bzip2: fix typoJes Cok
Remove redundant "this". Change-Id: Ia845e44119bf5ba1862f62da335466219a37c325 GitHub-Last-Rev: f0cdaf03285da8b053b71cc2911f5ab5beff580b GitHub-Pull-Request: golang/go#60807 Reviewed-on: https://go-review.googlesource.com/c/go/+/503655 Reviewed-by: Ian Lance Taylor <iant@google.com> Reviewed-by: Bryan Mills <bcmills@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Run-TryBot: Ian Lance Taylor <iant@google.com> Auto-Submit: Ian Lance Taylor <iant@google.com>
2023-06-14all: fix spelling errorsAlexander Yastrebov
Fix spelling errors discovered using https://github.com/codespell-project/codespell. Errors in data files and vendored packages are ignored. Change-Id: I83c7818222f2eea69afbd270c15b7897678131dc GitHub-Last-Rev: 3491615b1b82832cc0064f535786546e89aa6184 GitHub-Pull-Request: golang/go#60758 Reviewed-on: https://go-review.googlesource.com/c/go/+/502576 Auto-Submit: Michael Pratt <mpratt@google.com> Run-TryBot: Michael Pratt <mpratt@google.com> Reviewed-by: Ian Lance Taylor <iant@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Michael Pratt <mpratt@google.com>
2023-05-24compress/flate, archive/zip: reduce memory allocationsMaksim Meshkov
The existing implementation allocates a new 4KB buffer each time it opens flate-encoded file in a zip archive. This commit allows the flate reader to reuse the buffer on call Reset instead of allocating a new one. It is noticeable when a zip archive contains a huge amount of files, e.g. zip archive has 50_000 files, for each file 4KB buffer is allocated, so it is 200MB memory allocations. If files are read sequentially only one buffer is needed. Fixes #59774 Change-Id: Ib16336b101ba58e8f0f30a45dc5fd4eeebc801a1 GitHub-Last-Rev: f3f395b2ad95b7ad7ce9df6f5e49c7b6a0627627 GitHub-Pull-Request: golang/go#59775 Reviewed-on: https://go-review.googlesource.com/c/go/+/487675 Run-TryBot: Ian Lance Taylor <iant@golang.org> Auto-Submit: Ian Lance Taylor <iant@google.com> Run-TryBot: Ian Lance Taylor <iant@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@google.com> Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2023-02-27compress/zlib: use binary.BigEndian consistentlyJoe Tsai
One major reason to avoid binary.BigEndian is because the binary package includes a transitive dependency on reflect. See #54097. Given that writer.go already depends on the binary package, embrace use of it consistently where sensible. We should either embrace use of binary or fully avoid it. Change-Id: I5f2d27d0ed8cab5ac54be02362c7d33276dd4b9a Reviewed-on: https://go-review.googlesource.com/c/go/+/452176 Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Daniel Martí <mvdan@mvdan.cc> Run-TryBot: Joseph Tsai <joetsai@digital-static.net> Auto-Submit: Joseph Tsai <joetsai@digital-static.net> Reviewed-by: Ian Lance Taylor <iant@google.com> TryBot-Result: Gopher Robot <gobot@golang.org>
2022-11-18all: add missing periods in commentscui fliter
Change-Id: I69065f8adf101fdb28682c55997f503013a50e29 Reviewed-on: https://go-review.googlesource.com/c/go/+/449757 Auto-Submit: Ian Lance Taylor <iant@google.com> Reviewed-by: Joedian Reid <joedian@golang.org> Reviewed-by: Keith Randall <khr@google.com> Reviewed-by: Keith Randall <khr@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> Run-TryBot: Joedian Reid <joedian@golang.org> Run-TryBot: Ian Lance Taylor <iant@google.com> Reviewed-by: Ian Lance Taylor <iant@google.com>
2022-09-29compress: remove useless break statementcuiweixie
Change-Id: Ia2b376d134d4fd273924de2e4cdee9eba5a15c57 Reviewed-on: https://go-review.googlesource.com/c/go/+/436707 Reviewed-by: Ian Lance Taylor <iant@google.com> Run-TryBot: Ian Lance Taylor <iant@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Auto-Submit: Ian Lance Taylor <iant@google.com> TryBot-Result: Gopher Robot <gobot@golang.org>
2022-09-15compress/flate: update NewReader documentationJoe Tsai
Calling close is no longer necessary. It was was originally necessary to cleanup goroutines spawned to decompress the stream. This has not been the case since CL 4548079. Update the documentation to mention how it handles trailing data after the end of the DEFLATE stream. Change-Id: Ieacba264230560713b7b8d604665223fd096f4ec Reviewed-on: https://go-review.googlesource.com/c/go/+/430377 Run-TryBot: Joseph Tsai <joetsai@digital-static.net> Auto-Submit: Joseph Tsai <joetsai@digital-static.net> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@google.com> Reviewed-by: Than McIntosh <thanm@google.com>
2022-07-12compress/gzip: fix stack exhaustion bug in Reader.ReadTatiana Bradley
Replace recursion with iteration in Reader.Read to avoid stack exhaustion when there are a large number of files. Fixes CVE-2022-30631 Fixes #53168 Change-Id: I47d8afe3f2d40b0213ab61431df9b221794dbfe0 Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/1455673 Reviewed-by: Roland Shoemaker <bracewell@google.com> Reviewed-by: Julie Qiu <julieqiu@google.com> Reviewed-on: https://go-review.googlesource.com/c/go/+/417067 Reviewed-by: Heschi Kreinick <heschi@google.com> Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gopher Robot <gobot@golang.org>
2022-06-22compress/gzip: always close bodyReader in Example_compressingReaderIan Lance Taylor
For #53362 Fixes #53414 Change-Id: I352164e70c136eed210c7ee4ceba5dc631f81f94 Reviewed-on: https://go-review.googlesource.com/c/go/+/412955 Auto-Submit: Ian Lance Taylor <iant@google.com> Reviewed-by: Joseph Tsai <joetsai@digital-static.net> Reviewed-by: Ian Lance Taylor <iant@google.com> Run-TryBot: Ian Lance Taylor <iant@google.com> Reviewed-by: Alex Rakoczy <alex@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org>
2022-05-17all: fix spellingJohn Bampton
Change-Id: I68538a50c22b02cdb5aa2a889f9440fed7b94c54 GitHub-Last-Rev: aaac9e78340ac482e9cd1b506a035f271c29648c GitHub-Pull-Request: golang/go#52944 Reviewed-on: https://go-review.googlesource.com/c/go/+/406835 Reviewed-by: Bryan Mills <bcmills@google.com> Reviewed-by: Robert Griesemer <gri@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Run-TryBot: Bryan Mills <bcmills@google.com> Auto-Submit: Bryan Mills <bcmills@google.com>
2022-05-17compress/flate: remove var newHTobias Klauser
After CL 404696 hewH is no longer used outide the loop and val can be used inside the loop instead. This leads to another slight improvement in some benchmarks (only non-zero results reported): name old time/op new time/op delta Decode/Digits/Huffman/1e4-4 125µs ±40% 96µs ± 7% -22.72% (p=0.000 n=10+10) Decode/Digits/Huffman/1e5-4 1.02ms ± 4% 0.97ms ± 3% -4.29% (p=0.000 n=10+10) Decode/Digits/Huffman/1e6-4 10.5ms ± 3% 10.1ms ± 2% -3.34% (p=0.000 n=10+9) Decode/Digits/Speed/1e4-4 130µs ± 5% 119µs ± 3% -8.33% (p=0.000 n=10+10) Decode/Digits/Speed/1e5-4 1.32ms ± 3% 1.26ms ± 4% -4.14% (p=0.001 n=10+10) Decode/Digits/Speed/1e6-4 13.4ms ± 3% 12.8ms ± 3% -4.64% (p=0.000 n=10+10) Decode/Digits/Default/1e4-4 142µs ±19% 124µs ± 3% -12.75% (p=0.000 n=10+9) Decode/Digits/Default/1e5-4 1.33ms ± 4% 1.27ms ± 2% -4.45% (p=0.000 n=10+10) Decode/Digits/Compression/1e4-4 132µs ± 4% 126µs ± 3% -4.59% (p=0.000 n=10+10) Decode/Digits/Compression/1e5-4 1.31ms ± 4% 1.28ms ± 3% -2.12% (p=0.015 n=10+10) Decode/Digits/Compression/1e6-4 13.2ms ± 3% 12.8ms ± 4% -3.36% (p=0.000 n=10+10) Decode/Newton/Huffman/1e4-4 138µs ± 5% 128µs ± 3% -7.55% (p=0.000 n=10+9) Decode/Newton/Huffman/1e5-4 1.25ms ± 1% 1.23ms ± 3% -2.21% (p=0.027 n=8+10) Decode/Newton/Huffman/1e6-4 13.0ms ± 5% 12.2ms ± 5% -6.54% (p=0.000 n=10+10) Decode/Newton/Speed/1e4-4 128µs ± 3% 118µs ± 4% -7.34% (p=0.000 n=9+10) Decode/Newton/Speed/1e5-4 1.06ms ± 3% 1.02ms ± 3% -3.58% (p=0.001 n=9+10) Decode/Newton/Speed/1e6-4 10.4ms ± 4% 10.1ms ± 3% -3.15% (p=0.003 n=10+10) Decode/Newton/Compression/1e4-4 105µs ± 2% 108µs ± 5% +2.82% (p=0.043 n=9+10) Encode/Digits/Speed/1e5-4 1.65ms ± 2% 1.70ms ± 4% +2.77% (p=0.003 n=8+10) Encode/Newton/Default/1e6-4 58.0ms ± 2% 57.0ms ± 1% -1.59% (p=0.001 n=9+9) name old speed new speed delta Decode/Digits/Huffman/1e4-4 82.2MB/s ±30% 103.9MB/s ± 8% +26.38% (p=0.000 n=10+10) Decode/Digits/Huffman/1e5-4 98.5MB/s ± 4% 102.9MB/s ± 3% +4.46% (p=0.000 n=10+10) Decode/Digits/Huffman/1e6-4 95.6MB/s ± 3% 98.9MB/s ± 2% +3.44% (p=0.000 n=10+9) Decode/Digits/Speed/1e4-4 76.9MB/s ± 5% 83.8MB/s ± 3% +9.06% (p=0.000 n=10+10) Decode/Digits/Speed/1e5-4 75.8MB/s ± 3% 79.1MB/s ± 4% +4.34% (p=0.001 n=10+10) Decode/Digits/Speed/1e6-4 74.4MB/s ± 3% 78.0MB/s ± 3% +4.86% (p=0.000 n=10+10) Decode/Digits/Default/1e4-4 70.7MB/s ±17% 80.6MB/s ± 2% +13.93% (p=0.000 n=10+9) Decode/Digits/Default/1e5-4 75.4MB/s ± 4% 78.9MB/s ± 3% +4.60% (p=0.000 n=10+10) Decode/Digits/Compression/1e4-4 75.8MB/s ± 3% 79.4MB/s ± 3% +4.79% (p=0.000 n=10+10) Decode/Digits/Compression/1e5-4 76.5MB/s ± 4% 78.1MB/s ± 3% +2.15% (p=0.015 n=10+10) Decode/Digits/Compression/1e6-4 75.7MB/s ± 3% 78.3MB/s ± 4% +3.49% (p=0.000 n=10+10) Decode/Newton/Huffman/1e4-4 72.4MB/s ± 5% 78.3MB/s ± 3% +8.13% (p=0.000 n=10+9) Decode/Newton/Huffman/1e5-4 79.8MB/s ± 1% 81.6MB/s ± 3% +2.29% (p=0.026 n=8+10) Decode/Newton/Huffman/1e6-4 76.9MB/s ± 5% 82.2MB/s ± 5% +6.96% (p=0.000 n=10+10) Decode/Newton/Speed/1e4-4 78.4MB/s ± 3% 84.6MB/s ± 4% +7.92% (p=0.000 n=9+10) Decode/Newton/Speed/1e5-4 94.2MB/s ± 3% 97.7MB/s ± 3% +3.72% (p=0.001 n=9+10) Decode/Newton/Speed/1e6-4 96.0MB/s ± 4% 99.1MB/s ± 3% +3.24% (p=0.003 n=10+10) Decode/Newton/Compression/1e4-4 95.2MB/s ± 2% 92.6MB/s ± 5% -2.67% (p=0.043 n=9+10) Encode/Digits/Speed/1e5-4 60.6MB/s ± 2% 59.0MB/s ± 4% -2.66% (p=0.002 n=8+10) Encode/Newton/Default/1e6-4 17.3MB/s ± 2% 17.5MB/s ± 1% +1.60% (p=0.001 n=9+9) Change-Id: I833b008fe5b67cd17ffdfbec22a3e4b10fcddece Reviewed-on: https://go-review.googlesource.com/c/go/+/406754 Reviewed-by: Joseph Tsai <joetsai@digital-static.net> Reviewed-by: Ian Lance Taylor <iant@google.com> Auto-Submit: Tobias Klauser <tobias.klauser@gmail.com> Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com> Reviewed-by: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gopher Robot <gobot@golang.org>
2022-05-11compress/flate: remove compressor.hash fieldIan Lance Taylor
After CL 20929, the only remaining uses of d.hash are immediately after it is set. Benchmarks where benchstat reports a non-zero delta: name old time/op new time/op delta Decode/Newton/Huffman/1e5-12 982µs ± 6% 922µs ± 4% -6.12% (p=0.000 n=10+10) Decode/Newton/Huffman/1e6-12 8.14ms ± 2% 7.71ms ± 2% -5.19% (p=0.000 n=10+10) Encode/Digits/Huffman/1e5-12 241µs ± 0% 239µs ± 0% -0.62% (p=0.000 n=9+10) Encode/Digits/Speed/1e4-12 113µs ± 0% 114µs ± 0% +1.21% (p=0.000 n=9+8) Encode/Digits/Speed/1e5-12 1.04ms ± 0% 1.05ms ± 0% +0.22% (p=0.027 n=9+8) Encode/Digits/Speed/1e6-12 10.3ms ± 0% 10.3ms ± 0% +0.16% (p=0.036 n=9+8) Encode/Digits/Default/1e4-12 193µs ± 0% 195µs ± 1% +0.77% (p=0.010 n=8+8) Encode/Digits/Default/1e5-12 2.92ms ± 1% 2.93ms ± 0% +0.54% (p=0.002 n=8+8) Encode/Digits/Compression/1e4-12 193µs ± 0% 194µs ± 0% +0.67% (p=0.000 n=8+8) Encode/Digits/Compression/1e5-12 2.92ms ± 0% 2.93ms ± 0% +0.53% (p=0.002 n=8+8) Encode/Digits/Compression/1e6-12 31.0ms ± 1% 31.2ms ± 0% +0.80% (p=0.000 n=10+9) Encode/Newton/Huffman/1e4-12 34.8µs ± 1% 34.3µs ± 0% -1.31% (p=0.000 n=9+10) Encode/Newton/Huffman/1e5-12 294µs ± 0% 291µs ± 0% -1.09% (p=0.000 n=8+9) Encode/Newton/Huffman/1e6-12 2.91ms ± 0% 2.89ms ± 0% -0.89% (p=0.000 n=8+9) Encode/Newton/Speed/1e5-12 838µs ± 0% 836µs ± 0% -0.18% (p=0.015 n=8+8) Encode/Newton/Speed/1e6-12 8.35ms ± 0% 8.28ms ± 1% -0.89% (p=0.000 n=8+10) Encode/Newton/Default/1e5-12 3.55ms ± 0% 3.54ms ± 0% -0.23% (p=0.027 n=8+9) Encode/Newton/Default/1e6-12 37.4ms ± 0% 37.3ms ± 0% -0.18% (p=0.001 n=9+8) Encode/Newton/Compression/1e6-12 53.3ms ± 0% 53.1ms ± 0% -0.25% (p=0.010 n=10+9) name old speed new speed delta Decode/Newton/Huffman/1e5-12 102MB/s ± 6% 109MB/s ± 4% +6.48% (p=0.000 n=10+10) Decode/Newton/Huffman/1e6-12 123MB/s ± 2% 130MB/s ± 2% +5.49% (p=0.000 n=10+10) Encode/Digits/Huffman/1e5-12 416MB/s ± 0% 418MB/s ± 0% +0.62% (p=0.000 n=9+10) Encode/Digits/Speed/1e4-12 88.9MB/s ± 0% 87.8MB/s ± 0% -1.19% (p=0.000 n=9+8) Encode/Digits/Speed/1e5-12 95.7MB/s ± 0% 95.5MB/s ± 0% -0.22% (p=0.025 n=9+8) Encode/Digits/Speed/1e6-12 97.5MB/s ± 0% 97.3MB/s ± 0% -0.16% (p=0.026 n=9+8) Encode/Digits/Default/1e4-12 51.7MB/s ± 0% 51.3MB/s ± 1% -0.76% (p=0.007 n=8+8) Encode/Digits/Default/1e5-12 34.3MB/s ± 1% 34.1MB/s ± 0% -0.54% (p=0.001 n=8+8) Encode/Digits/Compression/1e4-12 51.8MB/s ± 0% 51.5MB/s ± 0% -0.66% (p=0.000 n=8+8) Encode/Digits/Compression/1e5-12 34.3MB/s ± 0% 34.1MB/s ± 0% -0.53% (p=0.002 n=8+8) Encode/Digits/Compression/1e6-12 32.3MB/s ± 1% 32.0MB/s ± 0% -0.79% (p=0.000 n=10+9) Encode/Newton/Huffman/1e4-12 288MB/s ± 1% 291MB/s ± 0% +1.33% (p=0.000 n=9+10) Encode/Newton/Huffman/1e5-12 340MB/s ± 0% 344MB/s ± 0% +1.10% (p=0.000 n=8+9) Encode/Newton/Huffman/1e6-12 343MB/s ± 0% 346MB/s ± 0% +0.90% (p=0.000 n=8+9) Encode/Newton/Speed/1e5-12 119MB/s ± 0% 120MB/s ± 0% +0.18% (p=0.014 n=8+8) Encode/Newton/Speed/1e6-12 120MB/s ± 0% 121MB/s ± 1% +0.90% (p=0.000 n=8+10) Encode/Newton/Default/1e5-12 28.2MB/s ± 0% 28.3MB/s ± 0% +0.23% (p=0.029 n=8+9) Encode/Newton/Default/1e6-12 26.7MB/s ± 0% 26.8MB/s ± 0% +0.20% (p=0.000 n=9+7) Encode/Newton/Compression/1e6-12 18.8MB/s ± 0% 18.8MB/s ± 0% +0.25% (p=0.012 n=10+9) Change-Id: I8ba0efcf5d42595d856082656b45f87bb3d7be0f Reviewed-on: https://go-review.googlesource.com/c/go/+/404696 Auto-Submit: Ian Lance Taylor <iant@google.com> Reviewed-by: Ian Lance Taylor <iant@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Joseph Tsai <joetsai@digital-static.net> Run-TryBot: Ian Lance Taylor <iant@google.com>
2022-05-08compress/flate: cancel redundant operationsHeisenberg
The assignment operation in the program seems to be redundant, the first judgment will continue to overwrite the previous value. The subsequent slicing operation will cut all the values without frequency. Change-Id: Id59fc36dd5bacfde881edaf0d9c1af5348286611 Reviewed-on: https://go-review.googlesource.com/c/go/+/244157 TryBot-Result: Gopher Robot <gobot@golang.org> Auto-Submit: Ian Lance Taylor <iant@google.com> Reviewed-by: Ian Lance Taylor <iant@google.com> Reviewed-by: David Chase <drchase@google.com> Reviewed-by: Joe Tsai <joetsai@google.com> Run-TryBot: Ian Lance Taylor <iant@google.com>
2022-05-02compress/flate: move idempotent close logic to compressorJoe Tsai
The compressor methods already have logic for handling a sticky error. Merge the logic from CL 136475 into that. This slightly changes the error message to be more sensible in the situation where it's returned by Flush. Updates #27741 Change-Id: Ie34cf3164d0fa6bd0811175ca467dbbcb3be1395 Reviewed-on: https://go-review.googlesource.com/c/go/+/403514 Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@google.com> Run-TryBot: Ian Lance Taylor <iant@google.com> Auto-Submit: Ian Lance Taylor <iant@google.com>
2022-05-02compress/flate: return error on closed stream writeGregory Man
Previously flate.Writer allowed writes after Close, and this behavior could lead to stream corruption. Fixes #27741 Change-Id: Iee1ac69f8199232f693dba77b275f7078257b582 Reviewed-on: https://go-review.googlesource.com/c/go/+/136475 Run-TryBot: Ian Lance Taylor <iant@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> Auto-Submit: Ian Lance Taylor <iant@google.com> Reviewed-by: Joseph Tsai <joetsai@digital-static.net> Reviewed-by: Carlos Amedee <carlos@golang.org> Reviewed-by: Ian Lance Taylor <iant@google.com> Run-TryBot: Ian Lance Taylor <iant@google.com>
2022-04-11all: gofmt main repoRuss Cox
[This CL is part of a sequence implementing the proposal #51082. The design doc is at https://go.dev/s/godocfmt-design.] Run the updated gofmt, which reformats doc comments, on the main repository. Vendored files are excluded. For #51082. Change-Id: I7332f099b60f716295fb34719c98c04eb1a85407 Reviewed-on: https://go-review.googlesource.com/c/go/+/384268 Reviewed-by: Jonathan Amsterdam <jba@google.com> Reviewed-by: Ian Lance Taylor <iant@golang.org>
2022-04-05all: replace `` and '' with “ (U+201C) and ” (U+201D) in doc commentsRuss Cox
go/doc in all its forms applies this replacement when rendering the comments. We are considering formatting doc comments, including doing this replacement as part of the formatting. Apply it to our source files ahead of time. For #51082. Change-Id: Ifcc1f5861abb57c5d14e7d8c2102dfb31b7a3a19 Reviewed-on: https://go-review.googlesource.com/c/go/+/384262 Trust: Russ Cox <rsc@golang.org> Run-TryBot: Russ Cox <rsc@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
2022-04-01all: fix various doc comment formatting nitsRuss Cox
A run of lines that are indented with any number of spaces or tabs format as a <pre> block. This commit fixes various doc comments that format badly according to that (standard) rule. For example, consider: // - List item. // Second line. // - Another item. Because the - lines are unindented, this is actually two paragraphs separated by a one-line <pre> block. This CL rewrites it to: // - List item. // Second line. // - Another item. Today, that will format as a single <pre> block. In a future release, we hope to format it as a bulleted list. Various other minor fixes as well, all in preparation for reformatting. For #51082. Change-Id: I95cf06040d4186830e571cd50148be3bf8daf189 Reviewed-on: https://go-review.googlesource.com/c/go/+/384257 Trust: Russ Cox <rsc@golang.org> Run-TryBot: Russ Cox <rsc@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org>
2022-03-30compress/zlib: tighten header CINFO checkNigel Tao
RFC 1950 section 2.2 "Data format" says "CINFO (Compression info)... For CM = 8... Values of CINFO above 7 are not allowed". Change-Id: Ibbc1213125c7dc045f09901ee7746660e90b5fcd Reviewed-on: https://go-review.googlesource.com/c/go/+/395734 Reviewed-by: Matthew Dempsky <mdempsky@google.com> Trust: Nigel Tao <nigeltao@golang.org>
2022-03-15compress/gzip: add example of compressing readerIan Lance Taylor
For #51092 Change-Id: If0a233651ac75f113569ddfffd056084f6092564 Reviewed-on: https://go-review.googlesource.com/c/go/+/389514 Trust: Ian Lance Taylor <iant@golang.org> Run-TryBot: Ian Lance Taylor <iant@golang.org> Reviewed-by: Joseph Tsai <joetsai@digital-static.net> TryBot-Result: Gopher Robot <gobot@golang.org>
2022-03-06all: fix some typosDan Kortschak
Change-Id: I7dfae0fc91c2d70873ec7ec920be7c0a4888153a Reviewed-on: https://go-review.googlesource.com/c/go/+/390175 Reviewed-by: Ian Lance Taylor <iant@golang.org> Reviewed-by: Daniel Martí <mvdan@mvdan.cc> Trust: Daniel Martí <mvdan@mvdan.cc>
2022-03-02compress/gzip: return unexpected EOF for certain truncated streamsPhil Bracikowski
For cases where RFC 1952 requires a field, the code returns the error io.ErrUnexpectedEOF except in two places: for the FNAME flag or the FCOMMENT flag. These flags expect a null-terminated string and readString may return an EOF if the Reader is truncated before a null byte is found. For consistency with parsing other parts of the header, this is converted to an unexpected EOF herein. Follow-up to CL 14832. Fixes #51417 Change-Id: I173283a6ae309e4a8e52fc15df404ce5db06eff1 GitHub-Last-Rev: 2e573cd961c0b1e4296fbde53097cf5723a2ca29 GitHub-Pull-Request: golang/go#51418 Reviewed-on: https://go-review.googlesource.com/c/go/+/389034 Reviewed-by: Joseph Tsai <joetsai@digital-static.net> Trust: Joseph Tsai <joetsai@digital-static.net> Trust: Ian Lance Taylor <iant@golang.org>
2022-01-13all: add a handful of fuzz targetsRoland Shoemaker
Adds simple fuzz targets to archive/tar, archive/zip, compress/gzip, encoding/json, image/jpeg, image/gif, and image/png. Second attempt, this time we don't use the archives in testdata when fuzzing archive/tar, since those are rather memory intensive, and were crashing a number of builders. Change-Id: I4828d64fa4763c0d8c980392a6578e4dfd956e13 Reviewed-on: https://go-review.googlesource.com/c/go/+/378174 Trust: Roland Shoemaker <roland@golang.org> Run-TryBot: Roland Shoemaker <roland@golang.org> Reviewed-by: Bryan Mills <bcmills@google.com> TryBot-Result: Gopher Robot <gobot@golang.org>
2022-01-12Revert "all: add a handful of fuzz targets"Bryan Mills
This reverts CL 352109. Reason for revert: causing OOM failures on several builders, and may cause OOMs for end users with small machines as well. Change-Id: I58308d09919969d5a6512ee5cee6aa5c4af6769b Reviewed-on: https://go-review.googlesource.com/c/go/+/377934 Trust: Bryan Mills <bcmills@google.com> Run-TryBot: Bryan Mills <bcmills@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Katie Hockman <katie@golang.org> Trust: Katie Hockman <katie@golang.org>
2022-01-12all: add a handful of fuzz targetsRoland Shoemaker
Adds simple fuzz targets to archive/tar, archive/zip, compress/gzip, encoding/json, image/jpeg, image/gif, and image/png. Change-Id: Ide1a8de88a9421e786eeeaea3bb93f41e0bae347 Reviewed-on: https://go-review.googlesource.com/c/go/+/352109 Trust: Katie Hockman <katie@golang.org> Reviewed-by: Katie Hockman <katie@golang.org> Trust: Roland Shoemaker <roland@golang.org> Run-TryBot: Roland Shoemaker <roland@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org>
2021-10-12compress/lzw: output a Clear code first, per GIF specNigel Tao
The TestStartsWithClearCode test is new, but if it existed beforehand, the want strings would be "\x81" and "Hi\x81" without a starting "\x80". Fixes #26108 Fixes #33748 Updates makeworld-the-better-one/didder#7 Updates nothings/stb#1222 Change-Id: I35ac0ed862ba6ee921ba9aee257bc19828abaa82 Reviewed-on: https://go-review.googlesource.com/c/go/+/354710 Trust: Nigel Tao <nigeltao@golang.org> Run-TryBot: Nigel Tao <nigeltao@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Andrew Gerrand <adg@golang.org>
2021-09-28compress/gzip: add missing licensejiahua wang
Change-Id: I80c2c6054f6a5be3555c72056159a47f927f9a44 Reviewed-on: https://go-review.googlesource.com/c/go/+/352409 Run-TryBot: Ian Lance Taylor <iant@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org> Trust: Heschi Kreinick <heschi@google.com>
2021-03-17compress/lzw: add Reset method to Reader and WriterAgniva De Sarker
We add a Reset method which clears any internal state of an encoder or a decoder to let it be reused again as a new Writer or Reader respectively. We also export the encoder and decoder structs, renaming them to be Reader and Writer, and we guarantee that the underlying types from the constructors will always be Reader and Writer respectively. Benchmark results by reusing the encoder: on cpu: Intel(R) Core(TM) i5-8265U CPU @ 1.60GHz name time/op Decoder/1e4-8 93.6µs ± 1% Decoder/1e-Reuse4-8 87.7µs ± 1% Decoder/1e5-8 877µs ± 1% Decoder/1e-Reuse5-8 860µs ± 3% Decoder/1e6-8 8.79ms ± 1% Decoder/1e-Reuse6-8 8.82ms ± 4% Encoder/1e4-8 168µs ± 2% Encoder/1e-Reuse4-8 160µs ± 1% Encoder/1e5-8 1.64ms ± 1% Encoder/1e-Reuse5-8 1.61ms ± 2% Encoder/1e6-8 16.2ms ± 6% Encoder/1e-Reuse6-8 15.8ms ± 2% name speed Decoder/1e4-8 107MB/s ± 1% Decoder/1e-Reuse4-8 114MB/s ± 1% Decoder/1e5-8 114MB/s ± 1% Decoder/1e-Reuse5-8 116MB/s ± 3% Decoder/1e6-8 114MB/s ± 1% Decoder/1e-Reuse6-8 113MB/s ± 5% Encoder/1e4-8 59.7MB/s ± 2% Encoder/1e-Reuse4-8 62.4MB/s ± 1% Encoder/1e5-8 61.1MB/s ± 1% Encoder/1e-Reuse5-8 62.0MB/s ± 2% Encoder/1e6-8 61.7MB/s ± 5% Encoder/1e-Reuse6-8 63.4MB/s ± 2% name alloc/op Decoder/1e4-8 21.8kB ± 0% Decoder/1e-Reuse4-8 50.0B ± 0% Decoder/1e5-8 21.8kB ± 0% Decoder/1e-Reuse5-8 70.4B ± 2% Decoder/1e6-8 21.9kB ± 0% Decoder/1e-Reuse6-8 271B ± 3% Encoder/1e4-8 77.9kB ± 0% Encoder/1e-Reuse4-8 4.17kB ± 0% Encoder/1e5-8 77.9kB ± 0% Encoder/1e-Reuse5-8 4.27kB ± 0% Encoder/1e6-8 77.9kB ± 0% Encoder/1e-Reuse6-8 5.22kB ± 0% name allocs/op Decoder/1e4-8 2.00 ± 0% Decoder/1e-Reuse4-8 1.00 ± 0% Decoder/1e5-8 2.00 ± 0% Decoder/1e-Reuse5-8 1.00 ± 0% Decoder/1e6-8 2.00 ± 0% Decoder/1e-Reuse6-8 1.00 ± 0% Encoder/1e4-8 3.00 ± 0% Encoder/1e-Reuse4-8 2.00 ± 0% Encoder/1e5-8 3.00 ± 0% Encoder/1e-Reuse5-8 2.00 ± 0% Encoder/1e6-8 3.00 ± 0% Encoder/1e-Reuse6-8 2.00 ± 0% Fixes #26535 Change-Id: Icde613fea6234a5bdce95f1e49910f5687e30b22 Reviewed-on: https://go-review.googlesource.com/c/go/+/273667 Trust: Agniva De Sarker <agniva.quicksilver@gmail.com> Trust: Joe Tsai <thebrokentoaster@gmail.com> Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2020-12-09all: update to use os.ReadFile, os.WriteFile, os.CreateTemp, os.MkdirTempRuss Cox
As part of #42026, these helpers from io/ioutil were moved to os. (ioutil.TempFile and TempDir became os.CreateTemp and MkdirTemp.) Update the Go tree to use the preferred names. As usual, code compiled with the Go 1.4 bootstrap toolchain and code vendored from other sources is excluded. ReadDir changes are in a separate CL, because they are not a simple search and replace. For #42026. Change-Id: If318df0216d57e95ea0c4093b89f65e5b0ababb3 Reviewed-on: https://go-review.googlesource.com/c/go/+/266365 Trust: Russ Cox <rsc@golang.org> Run-TryBot: Russ Cox <rsc@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
2020-12-02all: update to use filepath.WalkDir instead of filepath.WalkRuss Cox
Now that filepath.WalkDir is available, it is more efficient and should be used in place of filepath.Walk. Update the tree to reflect best practices. As usual, the code compiled with Go 1.4 during bootstrap is excluded. (In this CL, that's only cmd/dist.) For #42027. Change-Id: Ib0f7b1e43e50b789052f9835a63ced701d8c411c Reviewed-on: https://go-review.googlesource.com/c/go/+/267719 Trust: Russ Cox <rsc@golang.org> Run-TryBot: Russ Cox <rsc@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Rob Pike <r@golang.org>
2020-10-23all: fix quoting for compress/bzip2 and time's godocsubham sarkar
The existing usage of grave accent (`) and apostrophe (') at some places made godoc to ignore them and show it as it is. So, use both of the characters twice (consecutively) so that godoc can convert it to {left,right} double quotation mark. Fixes #41958 Change-Id: I64fd9b5fa34f416ad595009d09f5482e10bd8b4f Reviewed-on: https://go-review.googlesource.com/c/go/+/262397 Reviewed-by: Russ Cox <rsc@golang.org> Trust: Alberto Donizetti <alb.donizetti@gmail.com>
2020-10-20all: update references to symbols moved from io/ioutil to ioRuss Cox
The old ioutil references are still valid, but update our code to reflect best practices and get used to the new locations. Code compiled with the bootstrap toolchain (cmd/asm, cmd/dist, cmd/compile, debug/elf) must remain Go 1.4-compatible and is excluded. Also excluded vendored code. For #41190. Change-Id: I6d86f2bf7bc37a9d904b6cee3fe0c7af6d94d5b1 Reviewed-on: https://go-review.googlesource.com/c/go/+/263142 Trust: Russ Cox <rsc@golang.org> Run-TryBot: Russ Cox <rsc@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
2020-10-20all: update references to symbols moved from os to io/fsRuss Cox
The old os references are still valid, but update our code to reflect best practices and get used to the new locations. Code compiled with the bootstrap toolchain (cmd/asm, cmd/dist, cmd/compile, debug/elf) must remain Go 1.4-compatible and is excluded. For #41190. Change-Id: I8f9526977867c10a221e2f392f78d7dec073f1bd Reviewed-on: https://go-review.googlesource.com/c/go/+/243907 Trust: Russ Cox <rsc@golang.org> Run-TryBot: Russ Cox <rsc@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Rob Pike <r@golang.org>
2020-10-19compress/flate: fix corrupted outputKlaus Post
The fastest compression mode can pick up a false match for every 2GB of input data resulting in incorrectly decompressed data. Since matches are allowed to be up to and including at maxMatchOffset we must offset the buffer by an additional element to prevent the first 4 bytes to match after an out-of-reach value after shiftOffsets has been called. We offset by `maxMatchOffset + 1` so offset 0 in the table will now fail the `if offset > maxMatchOffset` in all cases. Fixes #41420 Change-Id: If1fbe01728e132b8a207e3f3f439edd832dcc710 GitHub-Last-Rev: 50fabab0da874c37543b139459a810e12e83cee2 GitHub-Pull-Request: golang/go#41477 Reviewed-on: https://go-review.googlesource.com/c/go/+/255879 Reviewed-by: Matthew Dempsky <mdempsky@google.com> Run-TryBot: Matthew Dempsky <mdempsky@google.com> TryBot-Result: Go Bot <gobot@golang.org> Trust: Joe Tsai <thebrokentoaster@gmail.com> Trust: Matthew Dempsky <mdempsky@google.com>
2020-10-16compress/flate: revert a goto for-loopDaniel Martí
In https://golang.org/cl/16528, a goto loop was chosen over a regular for loop since that would make the function inlinable. Thanks to the recent https://golang.org/cl/256459, for loops without a label can now be inlined. So we can undo the workaround and simplify the code. Also add the function to TestIntendedInlining, which passes both before and after the change, as expected. For #14768. Change-Id: Ie5df55a6bcb07c538ca331eef2f908807ff0b516 Reviewed-on: https://go-review.googlesource.com/c/go/+/263037 Trust: Daniel Martí <mvdan@mvdan.cc> Run-TryBot: Daniel Martí <mvdan@mvdan.cc> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Go Bot <gobot@golang.org>
2020-10-06compress/flate: remove unneeded zeroing of bytes array in ↵Dan Scales
(*huffmanBitWriter).reset There is no correctness reason to zero out the w.bytes array in (w *huffmanBitWriter).reset, since w.nbytes is correctly set to zero. The elements of the bytes array are always written sequentially, with nbytes indicating how many elements have been written, and are only read up to the current value of nybytes. We have a pprof profile of a web server that compresses its request/responses, and the zeroing in reset() is taking up 2.6% of the CPU time of the server (and could be causing more slowdowns elsewhere due to its effects on the cache). This overhead may be showing up especially because there are many request/responses that are all fairly small. I'm not sure if the zeroing of the bytes array was intended as extra protection of data across reset uses in the same program, but no protection is needed as long as the huffman_bit_writer code remains correct. Change-Id: I67f2b2f56cff9dcc38d8fc0aea885bb010aeedbf Reviewed-on: https://go-review.googlesource.com/c/go/+/258577 Run-TryBot: Dan Scales <danscales@google.com> Run-TryBot: Joe Tsai <joetsai@google.com> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Klaus Post <klauspost@gmail.com> Reviewed-by: Joe Tsai <joetsai@google.com> Trust: Joe Tsai <joetsai@google.com> Trust: Dan Scales <danscales@google.com>
2020-07-16compress/flate: fix another deflate Reset inconsistencyRuss Cox
While investigating #34121, fixed by CL 193605, I discovered another case where Reset was not quite resetting enough. This specific case is not a problem in Reset itself but rather that the Huffman bit writer in one code path is using uninitialized memory left over from a previous block, making the compression not choose the optimal compression method. Fixes #34121. Change-Id: I29245b28214d924e382f91e2c56b4b8a9b7da13d Reviewed-on: https://go-review.googlesource.com/c/go/+/243140 Run-TryBot: Russ Cox <rsc@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2020-07-16compress/flate: fix deflate Reset consistencyKlaus Post
Modify the overflow detection logic to shuffle the contents of the table to a lower offset to avoid leaking the effects of a previous use of compress.Writer past Reset calls. Fixes #34121 Change-Id: I9963eadfa5482881e7b7adbad4c2cae146b669ab GitHub-Last-Rev: 8b35798cdd4d5a901d6422647b12984d7e500ba3 GitHub-Pull-Request: golang/go#34128 Reviewed-on: https://go-review.googlesource.com/c/go/+/193605 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com> Run-TryBot: Joe Tsai <thebrokentoaster@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-11-21all: base64-encode binaries that will cause Apple notarization to failAndrew
Starting with macOS 10.15 (Catalina), Apple now requires all software distributed outside of the App Store to be notarized. Any binaries we distribute must abide by a strict set of requirements like code-signing and having a minimum target SDK of 10.9 (amongst others). Apple’s notarization service will recursively inspect archives looking to find notarization candidate binaries. If it finds a binary that does not meet the requirements or is unable to decompress an archive, it will reject the entire distribution. From cursory testing, it seems that the service uses content sniffing to determine file types, so changing the file extension will not work. There are some binaries and archives included in our distribution that are being detected by Apple’s service as potential candidates for notarization or decompression. As these are files used by tests and some are intentionally invalid, we don’t intend to ever make them compliant. As a workaround for this, we base64-encode any binaries or archives that Apple’s notarization service issues a warning for, as these warnings will become errors in January 2020. Updates #34986 Change-Id: I106fbb6227b61eb221755568f047ee11103c1680 Reviewed-on: https://go-review.googlesource.com/c/go/+/208118 Run-TryBot: Andrew Bonventre <andybons@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-09-14compress/lzw: clarify code invariantsNigel Tao
This follows on from https://go-review.googlesource.com/c/go/+/191358 which was submitted as a comment-only change. Benchmarks don't show any significant change: compress/lzw name old speed new speed delta Decoder/1e4-56 92.8MB/s ± 1% 92.7MB/s ± 1% ~ (p=1.000 n=5+5) Decoder/1e5-56 100MB/s ± 1% 100MB/s ± 1% ~ (p=0.746 n=5+5) Decoder/1e6-56 101MB/s ± 1% 101MB/s ± 1% ~ (p=0.381 n=5+5) image/gif name old speed new speed delta Decode-56 63.2MB/s ± 1% 63.2MB/s ± 1% ~ (p=0.690 n=5+5) Change-Id: Ic36b5410cb06ca258da32e40da1f1ff6c44cff86 Reviewed-on: https://go-review.googlesource.com/c/go/+/194938 Reviewed-by: Bryan C. Mills <bcmills@google.com> Run-TryBot: Bryan C. Mills <bcmills@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-09-12compress/lzw: fix comment re high-code invariantNigel Tao
The listed invariant, while technically true, was misleading, and the invariant can be tightened. We never actually get to (d.hi == d.overflow), due to the "d.hi--" line in the decoder.decode method. This is a comment-only commit, changing the comment to match the code. A follow-up commit could restore the comment, changing the code to match the original intented invariant. But the first step is to have the comment and the code say the same thing. Change-Id: Ifc9f78d5060454fc107af9be298026bf3043d400 Reviewed-on: https://go-review.googlesource.com/c/go/+/191358 Reviewed-by: Daniel Martí <mvdan@mvdan.cc> Reviewed-by: Bryan C. Mills <bcmills@google.com> Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-08-28compress/flate: improve deflate performance by register allocating the indexNao YONASHIRO
Use local index variable to help the go compiler use a register to for the hash index instead of continuous memory read and write operations. compress/flate: Encode/Digits/Huffman/1e4-4 35.3µs ± 1% 32.7µs ± 0% -7.48% (p=0.000 n=17+19) Encode/Digits/Huffman/1e5-4 330µs ± 0% 312µs ± 0% -5.55% (p=0.000 n=17+18) Encode/Digits/Huffman/1e6-4 3.30ms ± 0% 3.12ms ± 0% -5.64% (p=0.000 n=18+19) Encode/Digits/Speed/1e4-4 157µs ± 0% 156µs ± 0% -0.41% (p=0.000 n=17+19) Encode/Digits/Speed/1e5-4 1.46ms ± 0% 1.46ms ± 1% ~ (p=0.478 n=20+19) Encode/Digits/Speed/1e6-4 14.4ms ± 0% 14.4ms ± 0% ~ (p=0.835 n=19+20) Encode/Digits/Default/1e4-4 309µs ± 0% 310µs ± 0% +0.23% (p=0.000 n=19+17) Encode/Digits/Default/1e5-4 4.76ms ± 0% 4.76ms ± 0% ~ (p=0.297 n=19+19) Encode/Digits/Default/1e6-4 51.0ms ± 0% 51.0ms ± 1% ~ (p=0.233 n=18+19) Encode/Digits/Compression/1e4-4 309µs ± 0% 310µs ± 0% +0.21% (p=0.000 n=17+20) Encode/Digits/Compression/1e5-4 4.76ms ± 0% 4.76ms ± 0% ~ (p=0.749 n=20+19) Encode/Digits/Compression/1e6-4 50.9ms ± 0% 50.9ms ± 0% ~ (p=0.499 n=18+19) Encode/Newton/Huffman/1e4-4 51.9µs ± 0% 48.0µs ± 0% -7.61% (p=0.000 n=19+19) Encode/Newton/Huffman/1e5-4 396µs ± 0% 377µs ± 0% -4.79% (p=0.000 n=18+19) Encode/Newton/Huffman/1e6-4 3.95ms ± 0% 3.74ms ± 0% -5.21% (p=0.000 n=20+17) Encode/Newton/Speed/1e4-4 155µs ± 0% 154µs ± 0% -0.67% (p=0.000 n=17+18) Encode/Newton/Speed/1e5-4 1.17ms ± 0% 1.16ms ± 0% -0.64% (p=0.000 n=20+16) Encode/Newton/Speed/1e6-4 11.6ms ± 0% 11.5ms ± 0% -0.63% (p=0.000 n=19+20) Encode/Newton/Default/1e4-4 347µs ± 0% 347µs ± 0% ~ (p=0.744 n=20+19) Encode/Newton/Default/1e5-4 5.06ms ± 0% 5.02ms ± 0% -0.77% (p=0.000 n=20+19) Encode/Newton/Default/1e6-4 53.3ms ± 1% 52.8ms ± 0% -0.91% (p=0.000 n=18+16) Encode/Newton/Compression/1e4-4 351µs ± 0% 351µs ± 0% ~ (p=0.277 n=20+20) Encode/Newton/Compression/1e5-4 6.90ms ± 0% 6.85ms ± 0% -0.61% (p=0.000 n=19+18) Encode/Newton/Compression/1e6-4 73.2ms ± 0% 72.8ms ± 0% -0.52% (p=0.000 n=18+18) name old speed new speed delta Encode/Digits/Huffman/1e4-4 283MB/s ± 1% 306MB/s ± 0% +8.09% (p=0.000 n=17+19) Encode/Digits/Huffman/1e5-4 303MB/s ± 0% 321MB/s ± 0% +5.87% (p=0.000 n=18+18) Encode/Digits/Huffman/1e6-4 303MB/s ± 0% 321MB/s ± 0% +5.98% (p=0.000 n=18+19) Encode/Digits/Speed/1e4-4 63.9MB/s ± 0% 64.2MB/s ± 0% +0.41% (p=0.000 n=17+19) Encode/Digits/Speed/1e5-4 68.5MB/s ± 0% 68.4MB/s ± 1% ~ (p=0.481 n=20+19) Encode/Digits/Speed/1e6-4 69.4MB/s ± 0% 69.3MB/s ± 0% ~ (p=0.712 n=19+20) Encode/Digits/Default/1e4-4 32.3MB/s ± 0% 32.3MB/s ± 0% -0.23% (p=0.000 n=19+17) Encode/Digits/Default/1e5-4 21.0MB/s ± 0% 21.0MB/s ± 0% ~ (p=0.460 n=19+19) Encode/Digits/Default/1e6-4 19.6MB/s ± 0% 19.6MB/s ± 1% ~ (p=0.180 n=18+19) Encode/Digits/Compression/1e4-4 32.3MB/s ± 0% 32.3MB/s ± 0% -0.21% (p=0.000 n=17+20) Encode/Digits/Compression/1e5-4 21.0MB/s ± 0% 21.0MB/s ± 0% ~ (p=0.700 n=20+19) Encode/Digits/Compression/1e6-4 19.6MB/s ± 0% 19.6MB/s ± 0% ~ (p=0.486 n=18+19) Encode/Newton/Huffman/1e4-4 193MB/s ± 0% 208MB/s ± 0% +8.23% (p=0.000 n=19+19) Encode/Newton/Huffman/1e5-4 252MB/s ± 0% 265MB/s ± 0% +5.04% (p=0.000 n=18+19) Encode/Newton/Huffman/1e6-4 253MB/s ± 0% 267MB/s ± 0% +5.49% (p=0.000 n=20+17) Encode/Newton/Speed/1e4-4 64.5MB/s ± 0% 65.0MB/s ± 0% +0.67% (p=0.000 n=17+18) Encode/Newton/Speed/1e5-4 85.7MB/s ± 0% 86.3MB/s ± 0% +0.65% (p=0.000 n=20+16) Encode/Newton/Speed/1e6-4 86.2MB/s ± 0% 86.7MB/s ± 0% +0.63% (p=0.000 n=19+20) Encode/Newton/Default/1e4-4 28.9MB/s ± 0% 28.9MB/s ± 0% ~ (p=0.840 n=20+19) Encode/Newton/Default/1e5-4 19.8MB/s ± 0% 19.9MB/s ± 0% +0.78% (p=0.000 n=20+19) Encode/Newton/Default/1e6-4 18.8MB/s ± 1% 18.9MB/s ± 0% +0.93% (p=0.000 n=18+16) Encode/Newton/Compression/1e4-4 28.5MB/s ± 0% 28.5MB/s ± 0% ~ (p=0.244 n=20+20) Encode/Newton/Compression/1e5-4 14.5MB/s ± 0% 14.6MB/s ± 0% +0.61% (p=0.000 n=19+18) Encode/Newton/Compression/1e6-4 13.7MB/s ± 0% 13.7MB/s ± 0% +0.53% (p=0.000 n=18+18) image/png: name old time/op new time/op delta EncodeGray-4 2.16ms ± 1% 1.85ms ± 1% -14.17% (p=0.000 n=86+91) EncodeGrayWithBufferPool-4 1.99ms ± 0% 1.69ms ± 0% -15.09% (p=0.000 n=97+94) EncodeNRGBOpaque-4 6.51ms ± 1% 5.62ms ± 1% -13.66% (p=0.000 n=90+92) EncodeNRGBA-4 7.33ms ± 1% 6.12ms ± 1% -16.49% (p=0.000 n=89+90) EncodePaletted-4 5.10ms ± 1% 4.96ms ± 1% -2.76% (p=0.000 n=90+87) EncodeRGBOpaque-4 6.51ms ± 1% 5.63ms ± 1% -13.49% (p=0.000 n=94+87) EncodeRGBA-4 24.3ms ± 2% 23.0ms ± 0% -5.23% (p=0.000 n=91+89) name old speed new speed delta EncodeGray-4 142MB/s ± 1% 166MB/s ± 1% +16.50% (p=0.000 n=86+91) EncodeGrayWithBufferPool-4 154MB/s ± 0% 182MB/s ± 0% +17.78% (p=0.000 n=97+94) EncodeNRGBOpaque-4 189MB/s ± 1% 219MB/s ± 1% +15.82% (p=0.000 n=90+93) EncodeNRGBA-4 168MB/s ± 1% 201MB/s ± 1% +19.75% (p=0.000 n=89+90) EncodePaletted-4 60.3MB/s ± 1% 62.0MB/s ± 1% +2.84% (p=0.000 n=90+87) EncodeRGBOpaque-4 189MB/s ± 1% 218MB/s ± 1% +15.60% (p=0.000 n=94+87) EncodeRGBA-4 50.6MB/s ± 2% 53.4MB/s ± 0% +5.51% (p=0.000 n=91+89) Change-Id: Ifed4486a7ba19a26abe5cbf2142f15cc7464e84f Reviewed-on: https://go-review.googlesource.com/c/go/+/187837 Run-TryBot: Daniel Martí <mvdan@mvdan.cc> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
7apu2bq3rhoylwmucxizk54afw5jyda7pho4j5zdcqpwkufke7zdzwad.onion