New subject: [ruby-core:114947] [Ruby master Bug#19909] s390x zlib different deflate algorithm producing a different compressed byte stream causing test failures

2 Oct 2023

Issue #19909 has been reported by jaruga (Jun Aruga).

----------------------------------------
Bug #19909: s390x zlib different deflate algorithm producing a different compressed byte stream causing test failures
https://bugs.ruby-lang.org/issues/19909

* Author: jaruga (Jun Aruga)
* Status: Open
* Priority: Normal
* Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN
----------------------------------------
I faced the test failures ruby/zlib (zlib gem) in `make test-all` and ruby/spec (`make test-spec`) in Ubuntu version jammy (22.04) s390x. I intend to manage this issue on this ticket, and giving the space for users to comment about the issue.

## The test failures

First, below are the test failures by `make test-all` and `make test-spec` on Ubuntu jammy s390x.

Here is the used zlib version.

```
$ dpkg -s "$(dpkg -S /usr/include/zlib.h | cut -d ":" -f 1)" | grep ^Version
Version: 1:1.2.11.dfsg-2ubuntu9.2
```

```
+ make -s test-all -o exts TESTOPTS=-j3 -q --tty=no  RUBYOPT=-w
generating enc.mk
making enc
making srcs under enc
generating transdb.h
transdb.h unchanged
making trans
making encs
Run options: 
  --seed=42873
  "--ruby=./miniruby -I../lib -I. -I.ext/common  ../tool/runruby.rb --extout=.ext  -- --disable-gems"
  --excludes-dir=../test/.excludes
  --name=!/memory_leak/
  -j3
  -q
  --tty=no

# Running tests:


Retrying...

  1) Failure:
TestZlibDeflate#test_deflate_chunked [/home/travis/build/junaruga/ruby/test/zlib/test_zlib.rb:66]:
<7253> expected but was
<8325>.

  2) Failure:
TestZlibDeflate#test_deflate_chunked_break [/home/travis/build/junaruga/ruby/test/zlib/test_zlib.rb:92]:
<3632> expected but was
<4702>.

  3) Failure:
TestZlibGzipReader#test_unused2 [/home/travis/build/junaruga/ruby/test/zlib/test_zlib.rb:968]:
<24> expected but was
<23>.

  4) Failure:
TestZlib#test_gzip [/home/travis/build/junaruga/ruby/test/zlib/test_zlib.rb:1419]:
<"\x1F\x8B\b\x00\x00\x00\x00\x00\x00\xFFK\xCB\xCF\a\x00!es\x8C\x03\x00\x00\x00"> expected but was
<"\x1F\x8B\b\x00\x00\x00\x00\x00\x00\xFFJ\xCB\xCF\a\f\x00!es\x8C\x03\x00\x00\x00">.

  5) Failure:
TestZlib#test_deflate_stream [/home/travis/build/junaruga/ruby/test/zlib/test_zlib.rb:1411]:
<20016> expected but was
<21085>.

Finished tests in 290.303048s, 88.4145 tests/s, 21049.4999 assertions/s.
25667 tests, 6110734 assertions, 5 failures, 0 errors, 172 skips
```

```
$SETARCH make -s test-spec
...
1)
Zlib::Deflate.deflate deflates some data FAILED
Expected 
"x\x9Cc\x80\x03\x00\x00
\x00\x01" == 
"x\x9Cc`\x80\x01\x00\x00
\x00\x01"
to be truthy but was false
/home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate/deflate_spec.rb:10:in `block (2 levels) in <top (required)>'
/home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate/deflate_spec.rb:4:in `<top (required)>'

2)
Zlib::Deflate.deflate deflates lots of data FAILED
Expected "x\x9Cc\x18\xE1`\xA4\x83\x91\x0Eh\rF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0E\x06;\x00\x00\x80\x00\x00\x01" == 
"x\x9C\xED\xC1\x01\x01\x00\x00\x00\x80\x90\xFE\xAF\xEE\b
\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x18\x80\x00\x00\x01"
to be truthy but was false
/home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate/deflate_spec.rb:18:in `block (2 levels) in <top (required)>'
/home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate/deflate_spec.rb:4:in `<top (required)>'

3)
Zlib::Deflate.deflate deflates chunked data FAILED
Expected 21085 == 20016
to be truthy but was false
/home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate/deflate_spec.rb:32:in `block (2 levels) in <top (required)>'
/home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate/deflate_spec.rb:4:in `<top (required)>'

4)
Zlib::Deflate#deflate deflates some data FAILED
Expected 
"x\x9Cc\x80\x03\x00\x00
\x00\x01" == 
"x\x9Cc`\x80\x01\x00\x00
\x00\x01"
to be truthy but was false
/home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate/deflate_spec.rb:47:in `block (2 levels) in <top (required)>'
/home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate/deflate_spec.rb:36:in `<top (required)>'

5)
Zlib::Deflate#deflate deflates lots of data FAILED
Expected "x\x9Cc\x18\xE1`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x81\x06#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\f4\x00\x00\x80\x00\x00\x01" == 
"x\x9C\xED\xC1\x01\x01\x00\x00\x00\x80\x90\xFE\xAF\xEE\b
\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x18\x80\x00\x00\x01"
to be truthy but was false
/home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate/deflate_spec.rb:56:in `block (2 levels) in <top (required)>'
/home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate/deflate_spec.rb:36:in `<top (required)>'

6)
Zlib::Deflate#deflate without break deflates chunked data with final chunk FAILED
Expected 8325 == 7253
to be truthy but was false
/home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate/deflate_spec.rb:96:in `block (3 levels) in <top (required)>'
/home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate/deflate_spec.rb:68:in `<top (required)>'

7)
Zlib::Deflate#deflate with break deflates chunked data with final chunk FAILED
Expected 4702 == 3632
to be truthy but was false
/home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate/deflate_spec.rb:123:in `block (3 levels) in <top (required)>'
/home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate/deflate_spec.rb:68:in `<top (required)>'

8)
Zlib::Deflate#set_dictionary sets the dictionary FAILED
Expected "x\xBB\x14\xE1\x03\xCBJLJNIMK\xCF\xC8\xCC\x02\f\x00\x15\x86\x03\xF8" == "x\xBB\x14\xE1\x03\xCBKLJNIMK\xCF\xC8\xCC\x02\x00\x15\x86\x03\xF8"
to be truthy but was false
/home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate/set_dictionary_spec.rb:10:in `block (2 levels) in <top (required)>'
/home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate/set_dictionary_spec.rb:4:in `<top (required)>'

9)
Zlib.deflate deflates some data FAILED
Expected 
"x\x9C3\x84\x03\x00
\x91\x01\xEB" == 
"x\x9C34\x84\x01\x00
\x91\x01\xEB"
to be truthy but was false
/home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate_spec.rb:6:in `block (2 levels) in <top (required)>'
/home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate_spec.rb:4:in `<top (required)>'

10)
Zlib.gzip gzips the given string FAILED
Expected 
"24261MLJNI\x05\f\x00\x9D\x05\x00$
\x00\x00\x00" == 
"34261MLJNI\x05\x00\x9D\x05\x00$
\x00\x00\x00"
to be truthy but was false
/home/travis/build/junaruga/ruby/spec/ruby/library/zlib/gzip_spec.rb:13:in `block (2 levels) in <top (required)>'
/home/travis/build/junaruga/ruby/spec/ruby/library/zlib/gzip_spec.rb:4:in `<top (required)>'

11)
Zlib::GzipWriter#write writes some compressed data FAILED
Expected [50, 52, 50, 54, 49, 77, 76, 74, 78, 73, 5, 12, 0, 157, 5, 0, 36, 10, 0, 0, 0] == [51, 52, 50, 54, 49, 77, 76, 74, 78, 73, 5, 0, 157, 5, 0, 36, 10, 0, 0, 0]
to be truthy but was false
/home/travis/build/junaruga/ruby/spec/ruby/library/zlib/gzipwriter/write_spec.rb:19:in `block (2 levels) in <top (required)>'
/home/travis/build/junaruga/ruby/spec/ruby/library/zlib/gzipwriter/write_spec.rb:5:in `<top (required)>'

12)
Zlib::GzipWriter#write handles inputs of 2^23 bytes FAILED
Expected 34263 == 8176
to be truthy but was false
/home/travis/build/junaruga/ruby/spec/ruby/library/zlib/gzipwriter/write_spec.rb:34:in `block (2 levels) in <top (required)>'
/home/travis/build/junaruga/ruby/spec/ruby/library/zlib/gzipwriter/write_spec.rb:5:in `<top (required)>'

Finished in 114.477929 seconds

3715 files, 32609 examples, 190709 expectations, 12 failures, 0 errors, 0 tagged
```

## The root cause

This issue can happen with zlib library applying [the patch zlib/pull#410](https://github.com/madler/zlib/pull/410) that is to enable a different deflate algorithm producing a different compressed byte stream causing the test failures. I am not sure how a Ubuntu zlib deb package is developed in Ubuntu. However I was able to download and check the source file `zlib_1.2.11.dfsg-2ubuntu9.2.debian.tar.xz` on [the Ubuntu jammy-updates zlib deb package page](https://packages.ubuntu.com/jammy-updates/zlib1g).

debian/rules
```
...
 49 # s390x fails at compatibility.
 50 ifneq (,$(findstring $(DEB_HOST_ARCH), s390x))
 51 m32=-m31
 52 CFLAGS += -DDFLTCC_LEVEL_MASK=0x7e
 53 CONFIGURE_COMMON += --dfltcc
 54 CONFIGURE_HOST += --crc32-vx
 55 else
 56 m32=-m32
 57 endif
...
```

debian/patches/series
```
...
  4 410.patch
...
```

debian/patches/410.patch

```
From 992a7afc3edfa511dff0650d1c545b11bf64e655 Mon Sep 17 00:00:00 2001
From: Ilya Leoshkevich <iii@linux.ibm.com>
Date: Wed, 18 Jul 2018 13:14:07 +0200
Subject: [PATCH] Add support for IBM Z hardware-accelerated deflate
...
```

And the 410.patch is the [the patch zlib/pull#410](https://github.com/madler/zlib/pull/410) above.

## A reproducer

For convenience, here is a relatively small reproducer.

```
$ cat test/zlib/_test_zlib_test_deflate_chunked.rb
require 'zlib'

z = Zlib::Deflate.new
input = "\x01"
z.deflate(input) do |chunk|
end
final = z.finish
puts "final.length: #{final.length}"
```

Below is ok case.

```
$ ruby -I./lib test/zlib/_test_zlib_test_deflate_chunked.rb
final.length: 9
```

Below is the case with the zlib where the tests are failing in s390x.

```
$ ruby -I./lib test/zlib/_test_zlib_test_deflate_chunked.rb
final.length: 10
```

## Affected distributions and versions

Affected: at least Ubuntu jammy 22.04

I didn't see this issue in Ubuntu focal (20.04). The same `410.patch` is applied in both Ubuntu focal and jammy. But how to build is different. In Ubuntu jammy, the zlib is configured by `./configure --dfltcc`.

```
$ diff -u ubuntu_focal/zlib1g/debian/rules ubuntu_jammy/zlib1g/debian/rules
--- ubuntu_focal/zlib1g/debian/rules	2020-08-20 01:52:59.000000000 +0200
+++ ubuntu_jammy/zlib1g/debian/rules	2021-08-12 05:28:03.000000000 +0200
@@ -21,6 +21,9 @@
 LDFLAGS = `dpkg-buildflags --get LDFLAGS`
 EXTRA_MAKE =

+CONFIGURE_COMMON=--shared --prefix=/usr
+CONFIGURE_HOST=--libdir=\$${prefix}/lib/$(DEB_HOST_MULTIARCH)
+
 # binutils doesn't supply the prefixed version normally like GCC does so
 # we can't just unconditionally use DEB_HOST_GNU_TYPE-ar
 ifeq ($(DEB_HOST_GNU_TYPE),$(DEB_BUILD_GNU_TYPE))
@@ -46,8 +49,9 @@
 # s390x fails at compatibility.
 ifneq (,$(findstring $(DEB_HOST_ARCH), s390x))
 m32=-m31
-CFLAGS += -DDFLTCC
-EXTRA_MAKE += OBJA=dfltcc.o PIC_OBJA=dfltcc.lo
+CFLAGS += -DDFLTCC_LEVEL_MASK=0x7e
+CONFIGURE_COMMON += --dfltcc
+CONFIGURE_HOST += --crc32-vx
 else
 m32=-m32
 endif
@@ -95,7 +99,7 @@

 	if [ ! -f Makefile.stash ]; then cp Makefile Makefile.stash ; fi

-	AR=$(AR) CC="$(DEB_HOST_GNU_TYPE)-gcc" CFLAGS="$(CFLAGS)" LDFLAGS="$(LDFLAGS)" uname=GNU ./configure --shared --prefix=/usr --libdir=\$${prefix}/lib/$(DEB_HOST_MULTIARCH)
+	AR=$(AR) CC="$(DEB_HOST_GNU_TYPE)-gcc" CFLAGS="$(CFLAGS)" LDFLAGS="$(LDFLAGS)" uname=GNU ./configure $(CONFIGURE_COMMON) $(CONFIGURE_HOST)

 	touch $@

@@ -106,7 +110,7 @@
 	cp -r $(COPYLIST) debian/64
 	cd debian/64 && AR=$(AR) CC="$(DEB_HOST_GNU_TYPE)-gcc $(m64)" \
 	    CFLAGS="$(CFLAGS)" LDFLAGS="$(LDFLAGS)" \
-	    uname=GNU ./configure --shared --prefix=/usr  --libdir=\$${prefix}/usr/lib64
+	    uname=GNU ./configure $(CONFIGURE_COMMON) --libdir=\$${prefix}/usr/lib64
 	touch $@

 configure32-stamp: configure
@@ -116,7 +120,7 @@
 	cp -r $(COPYLIST) debian/32
 	cd debian/32 && AR=$(AR) CC="$(DEB_HOST_GNU_TYPE)-gcc $(m32)" \
 	    CFLAGS="$(CFLAGS)" LDFLAGS="$(LDFLAGS)" \
-	    uname=GNU ./configure --shared --prefix=/usr --libdir=\$${prefix}/usr/lib32
+	    uname=GNU ./configure $(CONFIGURE_COMMON) --libdir=\$${prefix}/usr/lib32
 	touch $@

 configuren32-stamp: configure
@@ -126,7 +130,7 @@
 	cp -r $(COPYLIST) debian/n32
 	cd debian/n32 && AR=$(AR) CC="$(DEB_HOST_GNU_TYPE)-gcc $(mn32)" \
 	    CFLAGS="$(CFLAGS)" LDFLAGS="$(LDFLAGS)" \
-	    uname=GNU ./configure --shared --prefix=/usr --libdir=\$${prefix}/usr/lib32
+	    uname=GNU ./configure $(CONFIGURE_COMMON) --libdir=\$${prefix}/usr/lib32
 	touch $@

 configurex32-stamp: configure
@@ -136,7 +140,7 @@
 	cp -r $(COPYLIST) debian/x32
 	cd debian/x32 && AR=$(AR) CC="$(DEB_HOST_GNU_TYPE)-gcc $(mx32)" \
 	    CFLAGS="$(CFLAGS)" LDFLAGS="$(LDFLAGS)" \
-	    uname=GNU ./configure --shared --prefix=/usr --libdir=\$${prefix}/usr/libx32
+	    uname=GNU ./configure $(CONFIGURE_COMMON) --libdir=\$${prefix}/usr/libx32
 	touch $@

 build: build-stamp $(EXTRA_BUILD)
```

I also didn't see this issue with [the zlib RPM package](https://src.fedoraproject.org/rpms/zlib) in Fedora rawhide (Fedora 40), though I see the zlib is configured by `./configure --dfltcc` applying the patch `zlib-*-IBM-Z-hw-accelerated-deflate.patch`. But when comparing the `zlib-1.2.13-IBM-Z-hw-accelerated-deflate.patch` in the zlib in Fedora rawhide (Fedora 40), and `410.patch` in Ubuntu jammy-updates. It is quite different.

## Workaround

The patch author commented that a workaround is to set environment variable `DFLTCC=0` to disable the different deflate algorithm at [zlib/issues#60](https://github.com/ruby/zlib/issues/60#issuecomment-1733149916).

## How I fixed the issue with the workaround

* ruby/zlib: https://github.com/ruby/zlib/pull/65
* ruby/spec: https://github.com/ruby/spec/pull/1088
* ruby/ruby: https://github.com/ruby/ruby/pull/8401
* RubyCI - s390x (Ubuntu), the server is actually Ubuntu jammy, and the cron job is running with the `DFLTCC=0` in the crontab.

## How can we do?

The [upstream patch zlib/pull#410](https://github.com/madler/zlib/pull/410) is not merged to the zlib repository yet. And I am not sure if we want to do like this, modifying the `common.mk` (and `configure.ac`).

```

diff --git a/common.mk b/common.mk
index b8ae911ef2..2605569443 100644
--- a/common.mk
+++ b/common.mk
@@ -982,6 +982,7 @@ test-spec: $(TEST_RUNNABLE)-test-spec
 yes-test-spec: yes-test-spec-precheck
        $(ACTIONS_GROUP)
        $(gnumake_recursive)$(Q) \
+       DFLTCC=0 \
        $(RUNRUBY) -r./$(arch)-fake -r$(tooldir)/rubyspec_temp \
                $(srcdir)/spec/mspec/bin/mspec run -B $(srcdir)/spec/default.mspec $(MSPECOPT) $(SPECOPTS)
        $(ACTIONS_ENDGROUP)
```

So, it seems to me that just documenting this issue and workaround somewhere in ruby/ruby is a good fix for this issue.





-- 
https://bugs.ruby-lang.org/

    

[ruby-core:114942] [Ruby master Bug#19909] s390x zlib different deflate algorithm producing a different compressed byte stream causing test failures

jaruga (Jun Aruga)

jaruga (Jun Aruga)

tags

participants (1)