Issue #19909 has been updated by jaruga (Jun Aruga). I sent PR to ruby/ruby to document this issue. https://github.com/ruby/ruby/pull/8589 ---------------------------------------- Bug #19909: s390x zlib different deflate algorithm producing a different compressed byte stream causing test failures https://bugs.ruby-lang.org/issues/19909#change-104818 * Author: jaruga (Jun Aruga) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- I faced the test failures ruby/zlib (zlib gem) in `make test-all` and ruby/spec (`make test-spec`) in Ubuntu version jammy (22.04) s390x. I intend to manage this issue on this ticket, and giving the space for users to comment about the issue. ## The test failures First, below are the test failures by `make test-all` and `make test-spec` on Ubuntu jammy s390x. Here is the used zlib version. ``` $ dpkg -s "$(dpkg -S /usr/include/zlib.h | cut -d ":" -f 1)" | grep ^Version Version: 1:1.2.11.dfsg-2ubuntu9.2 ``` ``` + make -s test-all -o exts TESTOPTS=-j3 -q --tty=no RUBYOPT=-w generating enc.mk making enc making srcs under enc generating transdb.h transdb.h unchanged making trans making encs Run options: --seed=42873 "--ruby=./miniruby -I../lib -I. -I.ext/common ../tool/runruby.rb --extout=.ext -- --disable-gems" --excludes-dir=../test/.excludes --name=!/memory_leak/ -j3 -q --tty=no # Running tests: Retrying... 1) Failure: TestZlibDeflate#test_deflate_chunked [/home/travis/build/junaruga/ruby/test/zlib/test_zlib.rb:66]: <7253> expected but was <8325>. 2) Failure: TestZlibDeflate#test_deflate_chunked_break [/home/travis/build/junaruga/ruby/test/zlib/test_zlib.rb:92]: <3632> expected but was <4702>. 3) Failure: TestZlibGzipReader#test_unused2 [/home/travis/build/junaruga/ruby/test/zlib/test_zlib.rb:968]: <24> expected but was <23>. 4) Failure: TestZlib#test_gzip [/home/travis/build/junaruga/ruby/test/zlib/test_zlib.rb:1419]: <"\x1F\x8B\b\x00\x00\x00\x00\x00\x00\xFFK\xCB\xCF\a\x00!es\x8C\x03\x00\x00\x00"> expected but was <"\x1F\x8B\b\x00\x00\x00\x00\x00\x00\xFFJ\xCB\xCF\a\f\x00!es\x8C\x03\x00\x00\x00">. 5) Failure: TestZlib#test_deflate_stream [/home/travis/build/junaruga/ruby/test/zlib/test_zlib.rb:1411]: <20016> expected but was <21085>. Finished tests in 290.303048s, 88.4145 tests/s, 21049.4999 assertions/s. 25667 tests, 6110734 assertions, 5 failures, 0 errors, 172 skips ``` ``` $SETARCH make -s test-spec ... 1) Zlib::Deflate.deflate deflates some data FAILED Expected "x\x9Cc\x80\x03\x00\x00 \x00\x01" == "x\x9Cc`\x80\x01\x00\x00 \x00\x01" to be truthy but was false /home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate/deflate_spec.rb:10:in `block (2 levels) in <top (required)>' /home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate/deflate_spec.rb:4:in `<top (required)>' 2) Zlib::Deflate.deflate deflates lots of data FAILED Expected "x\x9Cc\x18\xE1`\xA4\x83\x91\x0Eh\rF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0E\x06;\x00\x00\x80\x00\x00\x01" == "x\x9C\xED\xC1\x01\x01\x00\x00\x00\x80\x90\xFE\xAF\xEE\b \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x18\x80\x00\x00\x01" to be truthy but was false /home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate/deflate_spec.rb:18:in `block (2 levels) in <top (required)>' /home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate/deflate_spec.rb:4:in `<top (required)>' 3) Zlib::Deflate.deflate deflates chunked data FAILED Expected 21085 == 20016 to be truthy but was false /home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate/deflate_spec.rb:32:in `block (2 levels) in <top (required)>' /home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate/deflate_spec.rb:4:in `<top (required)>' 4) Zlib::Deflate#deflate deflates some data FAILED Expected "x\x9Cc\x80\x03\x00\x00 \x00\x01" == "x\x9Cc`\x80\x01\x00\x00 \x00\x01" to be truthy but was false /home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate/deflate_spec.rb:47:in `block (2 levels) in <top (required)>' /home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate/deflate_spec.rb:36:in `<top (required)>' 5) Zlib::Deflate#deflate deflates lots of data FAILED Expected "x\x9Cc\x18\xE1`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x81\x06#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\f4\x00\x00\x80\x00\x00\x01" == "x\x9C\xED\xC1\x01\x01\x00\x00\x00\x80\x90\xFE\xAF\xEE\b \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x18\x80\x00\x00\x01" to be truthy but was false /home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate/deflate_spec.rb:56:in `block (2 levels) in <top (required)>' /home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate/deflate_spec.rb:36:in `<top (required)>' 6) Zlib::Deflate#deflate without break deflates chunked data with final chunk FAILED Expected 8325 == 7253 to be truthy but was false /home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate/deflate_spec.rb:96:in `block (3 levels) in <top (required)>' /home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate/deflate_spec.rb:68:in `<top (required)>' 7) Zlib::Deflate#deflate with break deflates chunked data with final chunk FAILED Expected 4702 == 3632 to be truthy but was false /home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate/deflate_spec.rb:123:in `block (3 levels) in <top (required)>' /home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate/deflate_spec.rb:68:in `<top (required)>' 8) Zlib::Deflate#set_dictionary sets the dictionary FAILED Expected "x\xBB\x14\xE1\x03\xCBJLJNIMK\xCF\xC8\xCC\x02\f\x00\x15\x86\x03\xF8" == "x\xBB\x14\xE1\x03\xCBKLJNIMK\xCF\xC8\xCC\x02\x00\x15\x86\x03\xF8" to be truthy but was false /home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate/set_dictionary_spec.rb:10:in `block (2 levels) in <top (required)>' /home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate/set_dictionary_spec.rb:4:in `<top (required)>' 9) Zlib.deflate deflates some data FAILED Expected "x\x9C3\x84\x03\x00 \x91\x01\xEB" == "x\x9C34\x84\x01\x00 \x91\x01\xEB" to be truthy but was false /home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate_spec.rb:6:in `block (2 levels) in <top (required)>' /home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate_spec.rb:4:in `<top (required)>' 10) Zlib.gzip gzips the given string FAILED Expected "24261MLJNI\x05\f\x00\x9D\x05\x00$ \x00\x00\x00" == "34261MLJNI\x05\x00\x9D\x05\x00$ \x00\x00\x00" to be truthy but was false /home/travis/build/junaruga/ruby/spec/ruby/library/zlib/gzip_spec.rb:13:in `block (2 levels) in <top (required)>' /home/travis/build/junaruga/ruby/spec/ruby/library/zlib/gzip_spec.rb:4:in `<top (required)>' 11) Zlib::GzipWriter#write writes some compressed data FAILED Expected [50, 52, 50, 54, 49, 77, 76, 74, 78, 73, 5, 12, 0, 157, 5, 0, 36, 10, 0, 0, 0] == [51, 52, 50, 54, 49, 77, 76, 74, 78, 73, 5, 0, 157, 5, 0, 36, 10, 0, 0, 0] to be truthy but was false /home/travis/build/junaruga/ruby/spec/ruby/library/zlib/gzipwriter/write_spec.rb:19:in `block (2 levels) in <top (required)>' /home/travis/build/junaruga/ruby/spec/ruby/library/zlib/gzipwriter/write_spec.rb:5:in `<top (required)>' 12) Zlib::GzipWriter#write handles inputs of 2^23 bytes FAILED Expected 34263 == 8176 to be truthy but was false /home/travis/build/junaruga/ruby/spec/ruby/library/zlib/gzipwriter/write_spec.rb:34:in `block (2 levels) in <top (required)>' /home/travis/build/junaruga/ruby/spec/ruby/library/zlib/gzipwriter/write_spec.rb:5:in `<top (required)>' Finished in 114.477929 seconds 3715 files, 32609 examples, 190709 expectations, 12 failures, 0 errors, 0 tagged ``` ## The root cause This issue can happen with zlib library applying [the patch zlib/pull#410](https://github.com/madler/zlib/pull/410) that is to enable a different deflate algorithm producing a different compressed byte stream causing the test failures. I am not sure how a Ubuntu zlib deb package is developed in Ubuntu. However I was able to download and check the source file `zlib_1.2.11.dfsg-2ubuntu9.2.debian.tar.xz` on [the Ubuntu jammy-updates zlib deb package page](https://packages.ubuntu.com/jammy-updates/zlib1g). debian/rules ``` ... 49 # s390x fails at compatibility. 50 ifneq (,$(findstring $(DEB_HOST_ARCH), s390x)) 51 m32=-m31 52 CFLAGS += -DDFLTCC_LEVEL_MASK=0x7e 53 CONFIGURE_COMMON += --dfltcc 54 CONFIGURE_HOST += --crc32-vx 55 else 56 m32=-m32 57 endif ... ``` debian/patches/series ``` ... 4 410.patch ... ``` debian/patches/410.patch ``` From 992a7afc3edfa511dff0650d1c545b11bf64e655 Mon Sep 17 00:00:00 2001 From: Ilya Leoshkevich &lt;iii(a)linux.ibm.com&gt; Date: Wed, 18 Jul 2018 13:14:07 +0200 Subject: [PATCH] Add support for IBM Z hardware-accelerated deflate ... ``` And the 410.patch is the [the patch zlib/pull#410](https://github.com/madler/zlib/pull/410) above. ## A reproducer For convenience, here is a relatively small reproducer. ``` $ cat test/zlib/_test_zlib_test_deflate_chunked.rb require 'zlib' z = Zlib::Deflate.new input = "\x01" z.deflate(input) do |chunk| end final = z.finish puts "final.length: #{final.length}" ``` Below is ok case. ``` $ ruby -I./lib test/zlib/_test_zlib_test_deflate_chunked.rb final.length: 9 ``` Below is the case with the zlib where the tests are failing in s390x. ``` $ ruby -I./lib test/zlib/_test_zlib_test_deflate_chunked.rb final.length: 10 ``` ## Affected distributions and versions Affected: at least Ubuntu jammy 22.04 I didn't see this issue in Ubuntu focal (20.04). The same `410.patch` is applied in both Ubuntu focal and jammy. But how to build is different. In Ubuntu jammy, the zlib is configured by `./configure --dfltcc`. ``` $ diff -u ubuntu_focal/zlib1g/debian/rules ubuntu_jammy/zlib1g/debian/rules --- ubuntu_focal/zlib1g/debian/rules 2020-08-20 01:52:59.000000000 +0200 +++ ubuntu_jammy/zlib1g/debian/rules 2021-08-12 05:28:03.000000000 +0200 @@ -21,6 +21,9 @@ LDFLAGS = `dpkg-buildflags --get LDFLAGS` EXTRA_MAKE = +CONFIGURE_COMMON=--shared --prefix=/usr +CONFIGURE_HOST=--libdir=\$${prefix}/lib/$(DEB_HOST_MULTIARCH) + # binutils doesn't supply the prefixed version normally like GCC does so # we can't just unconditionally use DEB_HOST_GNU_TYPE-ar ifeq ($(DEB_HOST_GNU_TYPE),$(DEB_BUILD_GNU_TYPE)) @@ -46,8 +49,9 @@ # s390x fails at compatibility. ifneq (,$(findstring $(DEB_HOST_ARCH), s390x)) m32=-m31 -CFLAGS += -DDFLTCC -EXTRA_MAKE += OBJA=dfltcc.o PIC_OBJA=dfltcc.lo +CFLAGS += -DDFLTCC_LEVEL_MASK=0x7e +CONFIGURE_COMMON += --dfltcc +CONFIGURE_HOST += --crc32-vx else m32=-m32 endif @@ -95,7 +99,7 @@ if [ ! -f Makefile.stash ]; then cp Makefile Makefile.stash ; fi - AR=$(AR) CC="$(DEB_HOST_GNU_TYPE)-gcc" CFLAGS="$(CFLAGS)" LDFLAGS="$(LDFLAGS)" uname=GNU ./configure --shared --prefix=/usr --libdir=\$${prefix}/lib/$(DEB_HOST_MULTIARCH) + AR=$(AR) CC="$(DEB_HOST_GNU_TYPE)-gcc" CFLAGS="$(CFLAGS)" LDFLAGS="$(LDFLAGS)" uname=GNU ./configure $(CONFIGURE_COMMON) $(CONFIGURE_HOST) touch $@ @@ -106,7 +110,7 @@ cp -r $(COPYLIST) debian/64 cd debian/64 && AR=$(AR) CC="$(DEB_HOST_GNU_TYPE)-gcc $(m64)" \ CFLAGS="$(CFLAGS)" LDFLAGS="$(LDFLAGS)" \ - uname=GNU ./configure --shared --prefix=/usr --libdir=\$${prefix}/usr/lib64 + uname=GNU ./configure $(CONFIGURE_COMMON) --libdir=\$${prefix}/usr/lib64 touch $@ configure32-stamp: configure @@ -116,7 +120,7 @@ cp -r $(COPYLIST) debian/32 cd debian/32 && AR=$(AR) CC="$(DEB_HOST_GNU_TYPE)-gcc $(m32)" \ CFLAGS="$(CFLAGS)" LDFLAGS="$(LDFLAGS)" \ - uname=GNU ./configure --shared --prefix=/usr --libdir=\$${prefix}/usr/lib32 + uname=GNU ./configure $(CONFIGURE_COMMON) --libdir=\$${prefix}/usr/lib32 touch $@ configuren32-stamp: configure @@ -126,7 +130,7 @@ cp -r $(COPYLIST) debian/n32 cd debian/n32 && AR=$(AR) CC="$(DEB_HOST_GNU_TYPE)-gcc $(mn32)" \ CFLAGS="$(CFLAGS)" LDFLAGS="$(LDFLAGS)" \ - uname=GNU ./configure --shared --prefix=/usr --libdir=\$${prefix}/usr/lib32 + uname=GNU ./configure $(CONFIGURE_COMMON) --libdir=\$${prefix}/usr/lib32 touch $@ configurex32-stamp: configure @@ -136,7 +140,7 @@ cp -r $(COPYLIST) debian/x32 cd debian/x32 && AR=$(AR) CC="$(DEB_HOST_GNU_TYPE)-gcc $(mx32)" \ CFLAGS="$(CFLAGS)" LDFLAGS="$(LDFLAGS)" \ - uname=GNU ./configure --shared --prefix=/usr --libdir=\$${prefix}/usr/libx32 + uname=GNU ./configure $(CONFIGURE_COMMON) --libdir=\$${prefix}/usr/libx32 touch $@ build: build-stamp $(EXTRA_BUILD) ``` I also didn't see this issue with [the zlib RPM package](https://src.fedoraproject.org/rpms/zlib) in Fedora rawhide (Fedora 40), though I see the zlib is configured by `./configure --dfltcc` applying the patch `zlib-*-IBM-Z-hw-accelerated-deflate.patch`. But when comparing the `zlib-1.2.13-IBM-Z-hw-accelerated-deflate.patch` in the zlib in Fedora rawhide (Fedora 40), and `410.patch` in Ubuntu jammy-updates. It is quite different. ## Workaround The patch author commented that a workaround is to set environment variable `DFLTCC=0` to disable the different deflate algorithm at [zlib/issues#60](https://github.com/ruby/zlib/issues/60#issuecomment-1733149…. ## How I fixed the issue with the workaround * ruby/zlib: https://github.com/ruby/zlib/pull/65 * ruby/spec: https://github.com/ruby/spec/pull/1088 * ruby/ruby: https://github.com/ruby/ruby/pull/8401 * RubyCI - s390x (Ubuntu), the server is actually Ubuntu jammy, and the cron job is running with the `DFLTCC=0` in the crontab. ## How can we do? The [upstream patch zlib/pull#410](https://github.com/madler/zlib/pull/410) is not merged to the zlib repository yet. And I am not sure if we want to do like this, modifying the `common.mk` (and `configure.ac`). ``` diff --git a/common.mk b/common.mk index b8ae911ef2..2605569443 100644 --- a/common.mk +++ b/common.mk @@ -982,6 +982,7 @@ test-spec: $(TEST_RUNNABLE)-test-spec yes-test-spec: yes-test-spec-precheck $(ACTIONS_GROUP) $(gnumake_recursive)$(Q) \ + DFLTCC=0 \ $(RUNRUBY) -r./$(arch)-fake -r$(tooldir)/rubyspec_temp \ $(srcdir)/spec/mspec/bin/mspec run -B $(srcdir)/spec/default.mspec $(MSPECOPT) $(SPECOPTS) $(ACTIONS_ENDGROUP) ``` So, it seems to me that just documenting this issue and workaround somewhere in ruby/ruby is a good fix for this issue. -- https://bugs.ruby-lang.org/