Use zlib-ng instead of zlib #8500

nulano · 2024-10-25T19:20:37Z

Alternative to #8495 .

Changes proposed in this pull request:

Replace zlib with zlib-ng built with CMake in Windows builds. Building our own zlib-ng allows us to use the latest release since the zlib-ng 2.2.2 release is missing the x86_64 Windows build.

Add a flag to PIL.features for zlib-ng. python3 -m PIL.report now looks like this:

--------------------------------------------------------------------
Pillow 11.1.0.dev0
Python 3.11.9 (tags/v3.11.9:de54cf5, Apr  2 2024, 10:12:12) [MSC v.1938 64 bit (AMD64)]
--------------------------------------------------------------------
Python executable is C:\hostedtoolcache\windows\Python\3.11.9\x64\python.exe
System Python files loaded from C:\hostedtoolcache\windows\Python\3.11.9\x64
--------------------------------------------------------------------
Python Pillow modules loaded from C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\site-packages\PIL
Binary Pillow modules loaded from C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\site-packages\PIL
--------------------------------------------------------------------
--- PIL CORE support ok, compiled for 11.1.0.dev0
--- TKINTER support ok, loaded 8.6
--- FREETYPE2 support ok, loaded 2.13.3
--- LITTLECMS2 support ok, loaded 2.16
--- WEBP support ok, loaded 1.4.0
--- JPEG support ok, compiled for libjpeg-turbo 3.0.4
--- OPENJPEG (JPEG2000) support ok, loaded 2.5.2
--- ZLIB (PNG/ZIP) support ok, loaded 1.3.1.zlib-ng, compiled for zlib-ng 2.2.2
--- LIBTIFF support ok, loaded 4.6.0
--- RAQM (Bidirectional Text) support ok, loaded 0.10.1, fribidi 1.0.16, harfbuzz 10.0.1
--- LIBIMAGEQUANT (Quantization method) support ok, loaded 2.16.0
*** XCB (X protocol) support not installed
--------------------------------------------------------------------

I took inspiration from #8495 (comment) to make a benchmark with multiple compression values. It seems that zlib-ng usually produces a slightly larger file but in much less time during compression.

Script and results (click to expand)

import io
import os
import timeit

from PIL import Image

PATH = "Tests/images/hopper.png"  # or "Tests/images/effect_spread.png"
REPEAT = 1000

print(f"{Image.__version__ = }")
print(f"{Image.core.zlib_version = }")
print(f"Testing {PATH = } with {REPEAT} repetitions")

with open(PATH, "rb") as f:
    data_in = f.read()

reader = io.BytesIO(data_in)
writer = io.BytesIO()

img = Image.open(reader)
img.load()


def read_png():
    reader.seek(0, os.SEEK_SET)
    return Image.open(reader)


def write_png(compress_level):
    writer.seek(0, os.SEEK_SET)
    writer.truncate(0)
    img.save(writer, "PNG", compress_level=compress_level)


t = timeit.timeit(read_png, number=REPEAT * 10)
print(f"read PNG: time = {t:f} (sec)")

for compress_level in range(0, 10):
    t = timeit.timeit(lambda: write_png(compress_level), number=REPEAT)
    print(
        f"write PNG: time = {t:f} (sec); size = {len(writer.getvalue())} bytes; {compress_level = }"
    )

Image.open(io.BytesIO(writer.getvalue())).show()

# Image.__version__ = '11.1.0.dev0'
# Image.core.zlib_version = '1.3.1'
# Testing PATH = 'Tests/images/hopper.png' with 1000 repetitions
# read PNG: time = 0.157310 (sec)
# write PNG: time = 0.376219 (sec); size = 49353 bytes; compress_level = 0
# write PNG: time = 0.945004 (sec); size = 32403 bytes; compress_level = 1
# write PNG: time = 1.046024 (sec); size = 32156 bytes; compress_level = 2
# write PNG: time = 1.238888 (sec); size = 31792 bytes; compress_level = 3
# write PNG: time = 1.549707 (sec); size = 30715 bytes; compress_level = 4
# write PNG: time = 1.881609 (sec); size = 30529 bytes; compress_level = 5
# write PNG: time = 2.527365 (sec); size = 30343 bytes; compress_level = 6
# write PNG: time = 3.075476 (sec); size = 30277 bytes; compress_level = 7
# write PNG: time = 5.036960 (sec); size = 30183 bytes; compress_level = 8
# write PNG: time = 6.598194 (sec); size = 30166 bytes; compress_level = 9

# Image.__version__ = '11.1.0.dev0'
# Image.core.zlib_version = '1.3.1.zlib-ng'
# Testing PATH = 'Tests/images/hopper.png' with 1000 repetitions
# read PNG: time = 0.178209 (sec)
# write PNG: time = 0.382191 (sec); size = 49353 bytes; compress_level = 0
# write PNG: time = 0.549274 (sec); size = 40354 bytes; compress_level = 1
# write PNG: time = 0.770737 (sec); size = 31458 bytes; compress_level = 2
# write PNG: time = 0.934707 (sec); size = 31096 bytes; compress_level = 3
# write PNG: time = 0.977891 (sec); size = 30836 bytes; compress_level = 4
# write PNG: time = 1.101530 (sec); size = 30771 bytes; compress_level = 5
# write PNG: time = 1.212539 (sec); size = 30649 bytes; compress_level = 6
# write PNG: time = 1.383462 (sec); size = 30217 bytes; compress_level = 7
# write PNG: time = 1.946067 (sec); size = 30175 bytes; compress_level = 8
# write PNG: time = 2.788035 (sec); size = 30166 bytes; compress_level = 9

# Image.__version__ = '11.1.0.dev0'
# Image.core.zlib_version = '1.3.1'
# Testing PATH = 'Tests/images/effect_spread.png' with 1000 repetitions
# read PNG: time = 0.096424 (sec)
# write PNG: time = 0.410247 (sec); size = 49353 bytes; compress_level = 0
# write PNG: time = 1.194126 (sec); size = 42199 bytes; compress_level = 1
# write PNG: time = 1.164452 (sec); size = 42096 bytes; compress_level = 2
# write PNG: time = 1.182063 (sec); size = 41977 bytes; compress_level = 3
# write PNG: time = 1.404397 (sec); size = 41011 bytes; compress_level = 4
# write PNG: time = 1.531614 (sec); size = 40969 bytes; compress_level = 5
# write PNG: time = 1.720916 (sec); size = 40903 bytes; compress_level = 6
# write PNG: time = 1.806385 (sec); size = 40896 bytes; compress_level = 7
# write PNG: time = 1.914409 (sec); size = 40889 bytes; compress_level = 8
# write PNG: time = 1.936970 (sec); size = 40889 bytes; compress_level = 9

# Image.__version__ = '11.1.0.dev0'
# Image.core.zlib_version = '1.3.1.zlib-ng'
# Testing PATH = 'Tests/images/effect_spread.png' with 1000 repetitions
# read PNG: time = 0.088179 (sec)
# write PNG: time = 0.412940 (sec); size = 49353 bytes; compress_level = 0
# write PNG: time = 0.607201 (sec); size = 48241 bytes; compress_level = 1
# write PNG: time = 0.852157 (sec); size = 41287 bytes; compress_level = 2
# write PNG: time = 1.026876 (sec); size = 41168 bytes; compress_level = 3
# write PNG: time = 1.044624 (sec); size = 41134 bytes; compress_level = 4
# write PNG: time = 1.200349 (sec); size = 41120 bytes; compress_level = 5
# write PNG: time = 1.214951 (sec); size = 41118 bytes; compress_level = 6
# write PNG: time = 0.983906 (sec); size = 40895 bytes; compress_level = 7
# write PNG: time = 0.989961 (sec); size = 40889 bytes; compress_level = 8
# write PNG: time = 1.540352 (sec); size = 40889 bytes; compress_level = 9

Looking at the build logs, zlib-ng seems to have been properly detected by webp, libtiff, libpng, and freetype.

TODO:

Run the test suite on Windows arm64. I can use a borrowed M2 MacBook, but I won't have access to it at least until Monday.
We should also update the Linux...
...and macOS builds for consistency if we decide that we do want to go with zlib-ng.

dofuuz · 2024-10-26T01:19:30Z

zlib-ng v2.2.2 x86_64-compat was not released because of CI test error. It'll be released at next version again.
So, downloading pre-built binary is still a valid way.

Anyway, building with CMake is also a fine solution. I'll follow Pillow devs' decision and close my PR #8495 when it's decided.

hugovk · 2024-10-30T14:21:03Z

Here's a chart of the write times for different compression levels and images:

hopper.png: 1.3.1 zlib-ng (red) is much better than 1.3.1 (blue)
effect_spread.png: 1.3.1 zlib-ng (green) is noticeably better than 1.3.1 (yellow)

Summary: zlib-ng is much faster.

And file sizes:

compression level 0: same for all
compression level 1: 1.3.1 zlib-ng is much worse than 1.3.1
compression level 2-3: 1.3.1 zlib-ng is slightly better than 1.3.1
compression level 4-6: 1.3.1 zlib-ng is even more slightly worse than 1.3.1
compression level 7-9: about the same, zlib-ng ever so slightly better than 1.3.1

Summary: zlib-ng has same file sizes for compression level 0 and bigger files for compression level 1, but if you care about size, any higher compression level gives more or less the same file size (and zlib-ng is faster).

nulano · 2024-11-02T12:34:43Z

I've now checked that the Windows arm64 wheels pass the test suite on an MacBook Pro M2 Max (tested Python versions: 3.9.10, 3.10.11, 3.11.6, 3.12.0, 3.13.0).

Benchmark results comparing the Python 3.13 wheels of the 11.0.0 release and the pull request build (click to expand)

Image.__version__ = '11.0.0'
Image.core.zlib_version = '1.3.1'
Testing PATH = '../Pillow-11.0.0/Pillow-11.0.0/Tests/images/hopper.png' with 1000 repetitions
read PNG: time = 0.589200 (sec)
write PNG: time = 0.555782 (sec); size = 49353 bytes; compress_level = 0
write PNG: time = 1.114117 (sec); size = 32403 bytes; compress_level = 1
write PNG: time = 1.198036 (sec); size = 32156 bytes; compress_level = 2
write PNG: time = 1.402062 (sec); size = 31792 bytes; compress_level = 3
write PNG: time = 1.758227 (sec); size = 30715 bytes; compress_level = 4
write PNG: time = 2.065863 (sec); size = 30529 bytes; compress_level = 5
write PNG: time = 2.774447 (sec); size = 30343 bytes; compress_level = 6
write PNG: time = 3.542967 (sec); size = 30277 bytes; compress_level = 7
write PNG: time = 5.532757 (sec); size = 30183 bytes; compress_level = 8
write PNG: time = 7.352626 (sec); size = 30166 bytes; compress_level = 9

Image.__version__ = '11.1.0.dev0'
Image.core.zlib_version = '1.3.1.zlib-ng'
Testing PATH = '../Pillow-11.0.0/Pillow-11.0.0/Tests/images/hopper.png' with 1000 repetitions
read PNG: time = 0.569872 (sec)
write PNG: time = 0.529697 (sec); size = 49353 bytes; compress_level = 0
write PNG: time = 0.723248 (sec); size = 40354 bytes; compress_level = 1
write PNG: time = 0.942190 (sec); size = 31458 bytes; compress_level = 2
write PNG: time = 1.009014 (sec); size = 31096 bytes; compress_level = 3
write PNG: time = 1.053823 (sec); size = 30836 bytes; compress_level = 4
write PNG: time = 1.071575 (sec); size = 30771 bytes; compress_level = 5
write PNG: time = 1.170958 (sec); size = 30649 bytes; compress_level = 6
write PNG: time = 1.503532 (sec); size = 30217 bytes; compress_level = 7
write PNG: time = 2.005649 (sec); size = 30175 bytes; compress_level = 8
write PNG: time = 2.626763 (sec); size = 30166 bytes; compress_level = 9

Image.__version__ = '11.0.0'
Image.core.zlib_version = '1.3.1'
Testing PATH = '../Pillow-11.0.0/Pillow-11.0.0/Tests/images/effect_spread.png' with 1000 repetitions
read PNG: time = 0.234989 (sec)
write PNG: time = 0.550197 (sec); size = 49353 bytes; compress_level = 0
write PNG: time = 1.305417 (sec); size = 42199 bytes; compress_level = 1
write PNG: time = 1.334536 (sec); size = 42096 bytes; compress_level = 2
write PNG: time = 1.384041 (sec); size = 41977 bytes; compress_level = 3
write PNG: time = 1.622062 (sec); size = 41011 bytes; compress_level = 4
write PNG: time = 1.721847 (sec); size = 40969 bytes; compress_level = 5
write PNG: time = 1.927218 (sec); size = 40903 bytes; compress_level = 6
write PNG: time = 2.054222 (sec); size = 40896 bytes; compress_level = 7
write PNG: time = 2.198446 (sec); size = 40889 bytes; compress_level = 8
write PNG: time = 2.199249 (sec); size = 40889 bytes; compress_level = 9

Image.__version__ = '11.1.0.dev0'
Image.core.zlib_version = '1.3.1.zlib-ng'
Testing PATH = '../Pillow-11.0.0/Pillow-11.0.0/Tests/images/effect_spread.png' with 1000 repetitions
read PNG: time = 0.240964 (sec)
write PNG: time = 0.538218 (sec); size = 49353 bytes; compress_level = 0
write PNG: time = 0.799972 (sec); size = 48241 bytes; compress_level = 1
write PNG: time = 1.091412 (sec); size = 41287 bytes; compress_level = 2
write PNG: time = 1.177194 (sec); size = 41168 bytes; compress_level = 3
write PNG: time = 1.191972 (sec); size = 41134 bytes; compress_level = 4
write PNG: time = 1.195040 (sec); size = 41120 bytes; compress_level = 5
write PNG: time = 1.190594 (sec); size = 41118 bytes; compress_level = 6
write PNG: time = 1.280668 (sec); size = 40895 bytes; compress_level = 7
write PNG: time = 1.293338 (sec); size = 40889 bytes; compress_level = 8
write PNG: time = 1.738161 (sec); size = 40889 bytes; compress_level = 9

hugovk · 2024-11-18T12:21:09Z

Similar trend for time and identical numbers for size:

radarhere · 2024-11-25T09:55:58Z

winbuild/build_prepare.py

+        "license": "LICENSE.md",
+        "patch": {
+            r"CMakeLists.txt": {
+                "set_target_properties(zlib PROPERTIES OUTPUT_NAME zlibstatic${{SUFFIX}})": "set_target_properties(zlib PROPERTIES OUTPUT_NAME zlib)",  # noqa: E501


If zlib-ng creates zlibstatic.lib by default, then wouldn't it seem helpful to detect that in setup.py, in case a user had built it themselves? nulano#44

Oh, is the concern that zlib-ng might have been built without --zlib-compat?

radarhere · 2024-12-13T12:44:00Z

winbuild/build_prepare.py

        ],
        "headers": [r"z*.h"],
-        "libs": [r"*.lib"],
+        "libs": [r"zlib.lib"],


Suggested change

"libs": [r"zlib.lib"],

"libs": ["zlib.lib"],

hugovk · 2024-12-14T10:24:42Z

Here's the results of running the benchmark on macOS 15.1 M2 with Python 3.13.

Baseline: recent wheel from main: https://github.com/python-pillow/Pillow/actions/runs/12268369479 > dist-macos-latest-11.0-arm64.zip > pillow-11.1.0.dev0-cp313-cp313-macosx_11_0_arm64.whl

Test: recent wheel from this PR: https://github.com/python-pillow/Pillow/actions/runs/12328198837?pr=8500
dist-macos-latest-11.0-arm64.zip > pillow-11.1.0.dev0-cp313-cp313-macosx_11_0_arm64.whl

Results

# Image.__version__ = '11.1.0.dev0'
# Image.core.zlib_version = '1.3.1'
# Testing PATH = 'Tests/images/hopper.png' with 1000 repetitions
# read PNG: time = 0.121070 (sec)
# write PNG: time = 0.277860 (sec); size = 49353 bytes; compress_level = 0
# write PNG: time = 1.378973 (sec); size = 32403 bytes; compress_level = 1
# write PNG: time = 1.463843 (sec); size = 32156 bytes; compress_level = 2
# write PNG: time = 1.946944 (sec); size = 31792 bytes; compress_level = 3
# write PNG: time = 2.628619 (sec); size = 30715 bytes; compress_level = 4
# write PNG: time = 2.989835 (sec); size = 30529 bytes; compress_level = 5
# write PNG: time = 3.582502 (sec); size = 30343 bytes; compress_level = 6
# write PNG: time = 4.346017 (sec); size = 30277 bytes; compress_level = 7
# write PNG: time = 6.963706 (sec); size = 30183 bytes; compress_level = 8
# write PNG: time = 11.201260 (sec); size = 30166 bytes; compress_level = 9

# Image.__version__ = '11.1.0.dev0'
# Image.core.zlib_version = '1.3.1.zlib-ng'
# Testing PATH = 'Tests/images/hopper.png' with 1000 repetitions
# read PNG: time = 0.121355 (sec)
# write PNG: time = 0.243460 (sec); size = 49353 bytes; compress_level = 0
# write PNG: time = 0.484412 (sec); size = 40354 bytes; compress_level = 1
# write PNG: time = 0.582722 (sec); size = 31458 bytes; compress_level = 2
# write PNG: time = 0.647406 (sec); size = 31096 bytes; compress_level = 3
# write PNG: time = 0.849081 (sec); size = 30836 bytes; compress_level = 4
# write PNG: time = 0.699703 (sec); size = 30771 bytes; compress_level = 5
# write PNG: time = 0.916293 (sec); size = 30649 bytes; compress_level = 6
# write PNG: time = 1.426837 (sec); size = 30217 bytes; compress_level = 7
# write PNG: time = 1.855467 (sec); size = 30175 bytes; compress_level = 8
# write PNG: time = 2.772019 (sec); size = 30166 bytes; compress_level = 9

# Image.__version__ = '11.1.0.dev0'
# Image.core.zlib_version = '1.3.1'
# Testing PATH = 'Tests/images/effect_spread.png' with 1000 repetitions
# read PNG: time = 0.064432 (sec)
# write PNG: time = 0.271332 (sec); size = 49353 bytes; compress_level = 0
# write PNG: time = 1.590392 (sec); size = 42199 bytes; compress_level = 1
# write PNG: time = 1.749554 (sec); size = 42096 bytes; compress_level = 2
# write PNG: time = 2.188817 (sec); size = 41977 bytes; compress_level = 3
# write PNG: time = 2.496155 (sec); size = 41011 bytes; compress_level = 4
# write PNG: time = 2.278722 (sec); size = 40969 bytes; compress_level = 5
# write PNG: time = 2.514241 (sec); size = 40903 bytes; compress_level = 6
# write PNG: time = 2.503093 (sec); size = 40896 bytes; compress_level = 7
# write PNG: time = 2.616062 (sec); size = 40889 bytes; compress_level = 8
# write PNG: time = 2.614784 (sec); size = 40889 bytes; compress_level = 9

# Image.__version__ = '11.1.0.dev0'
# Image.core.zlib_version = '1.3.1.zlib-ng'
# Testing PATH = 'Tests/images/effect_spread.png' with 1000 repetitions
# read PNG: time = 0.062720 (sec)
# write PNG: time = 0.180120 (sec); size = 49353 bytes; compress_level = 0
# write PNG: time = 0.400339 (sec); size = 48241 bytes; compress_level = 1
# write PNG: time = 0.682549 (sec); size = 41287 bytes; compress_level = 2
# write PNG: time = 0.724561 (sec); size = 41168 bytes; compress_level = 3
# write PNG: time = 0.743060 (sec); size = 41134 bytes; compress_level = 4
# write PNG: time = 0.756581 (sec); size = 41120 bytes; compress_level = 5
# write PNG: time = 0.818945 (sec); size = 41118 bytes; compress_level = 6
# write PNG: time = 0.872731 (sec); size = 40895 bytes; compress_level = 7
# write PNG: time = 0.824981 (sec); size = 40889 bytes; compress_level = 8
# write PNG: time = 1.366639 (sec); size = 40889 bytes; compress_level = 9

Again, a similar trend for time and identical numbers for size:

hugovk

Thank you!

Let's also include this in the release notes.

winbuild: Replace zlib with zlib-ng built with CMake

3e0849b

nulano mentioned this pull request Oct 25, 2024

Replace Windows zlib with zlib-ng #8495

Closed

PIL.features: Add a compile-time zlib-ng feature flag and version number

7885066

nulano force-pushed the zlib-ng branch from 0a75993 to 7885066 Compare October 25, 2024 19:32

nulano added Build Windows Performance labels Oct 25, 2024

radarhere mentioned this pull request Oct 26, 2024

Only replace version suffix if zlib-ng is present nulano/Pillow#41

Merged

Merge branch 'main' into zlib-ng

c07e089

radarhere and others added 2 commits October 26, 2024 11:23

Only replace version suffix if zlib-ng is present

ed910a6

Merge branch 'main' into zlib-ng

5ab79bc

nulano mentioned this pull request Oct 28, 2024

Saving PNG images with PIL is 4 times slower than saving them with OpenCV #5986

Open

Merge branch 'main' into zlib-ng

3f5f3bb

radarhere mentioned this pull request Nov 20, 2024

Use zlib-ng in Linux wheels nulano/Pillow#42

Merged

Use zlib-ng on Linux

69c9a7f

nulano added the Linux label Nov 21, 2024

radarhere changed the title ~~Windows: Use zlib-ng instead of zlib~~ Use zlib-ng instead of zlib Nov 22, 2024

Merge branch 'main' into zlib-ng

dab795c

This was referenced Nov 22, 2024

Use zlib-ng on macOS nulano/Pillow#43

Closed

Detect zlibstatic.lib when building for Windows nulano/Pillow#44

Closed

radarhere reviewed Nov 25, 2024

View reviewed changes

Use zlib-ng on macOS

4986609

hugovk added the macOS label Nov 28, 2024

radarhere reviewed Dec 13, 2024

View reviewed changes

Merge branch 'main' into zlib-ng

01bc487

radarhere approved these changes Dec 14, 2024

View reviewed changes

hugovk approved these changes Dec 14, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use zlib-ng instead of zlib #8500

Use zlib-ng instead of zlib #8500

nulano commented Oct 25, 2024 •

edited by radarhere

Loading

dofuuz commented Oct 26, 2024 •

edited

Loading

hugovk commented Oct 30, 2024

nulano commented Nov 2, 2024 •

edited

Loading

hugovk commented Nov 18, 2024

radarhere Nov 25, 2024 •

edited

Loading

radarhere Dec 14, 2024

radarhere Dec 13, 2024

hugovk commented Dec 14, 2024

hugovk left a comment

Use zlib-ng instead of zlib #8500

Are you sure you want to change the base?

Use zlib-ng instead of zlib #8500

Conversation

nulano commented Oct 25, 2024 • edited by radarhere Loading

dofuuz commented Oct 26, 2024 • edited Loading

hugovk commented Oct 30, 2024

nulano commented Nov 2, 2024 • edited Loading

hugovk commented Nov 18, 2024

radarhere Nov 25, 2024 • edited Loading

Choose a reason for hiding this comment

radarhere Dec 14, 2024

Choose a reason for hiding this comment

radarhere Dec 13, 2024

Choose a reason for hiding this comment

hugovk commented Dec 14, 2024

hugovk left a comment

Choose a reason for hiding this comment

nulano commented Oct 25, 2024 •

edited by radarhere

Loading

dofuuz commented Oct 26, 2024 •

edited

Loading

nulano commented Nov 2, 2024 •

edited

Loading

radarhere Nov 25, 2024 •

edited

Loading