[ruby-core:112630] [Ruby master Bug#19468] Ruby 3.2: net/http sets UTF-8 encoding for binary responses
Issue #19468 has been reported by romuloceccon (Rômulo Ceccon). ---------------------------------------- Bug #19468: Ruby 3.2: net/http sets UTF-8 encoding for binary responses https://bugs.ruby-lang.org/issues/19468 * Author: romuloceccon (Rômulo Ceccon) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.1 (2023-02-22 revision 65ab2c1ef2) [x86_64-linux] * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- net/http on Ruby 3.2 has changed the encoding of binary responses from SSL connected hosts (non-SSL connections are not affected): ``` ruby # req.rb require 'openssl' require 'net/http' puts "openssl ext: #{OpenSSL::VERSION}" puts "openssl lib: #{OpenSSL::OPENSSL_VERSION}" puts "net-protocol: #{Net::Protocol::VERSION}" puts "net-http: #{Net::HTTP::VERSION}" puts Net::HTTP.get(URI(ARGV.first)).encoding ``` Ruby 3.1 (with updated net-protocol and net-http libs): ``` $ ruby -v ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [x86_64-linux] $ ruby req.rb https://www.gnu.org/software/gzip/manual/gzip.pdf openssl ext: 3.0.0 openssl lib: OpenSSL 1.1.1n 15 Mar 2022 net-protocol: 0.2.1 net-http: 0.3.2 ASCII-8BIT # <== CORRECT ``` Ruby 3.2 (latest git revision): ``` $ ruby -v ruby 3.2.1 (2023-02-22 revision 65ab2c1ef2) [x86_64-linux] $ ruby req.rb https://www.gnu.org/software/gzip/manual/gzip.pdf openssl ext: 3.1.0 openssl lib: OpenSSL 1.1.1n 15 Mar 2022 net-protocol: 0.2.1 net-http: 0.3.2 UTF-8 # <== WRONG ``` I've tracked the problem down to the SSL socket call at https://github.com/ruby/ruby/blob/9557c8edf2dcf18fdece066c596a71696b2f2b30/l.... The string returned has the encoding set to `ASCII-8BIT`, but `#ascii_only?` also always reports true, even when there are non-ascii bytes. This seems to be a bug, and is the probably cause of the change in behavior in net/http. On Ruby 3.1 concatenating the result of reading the SSL socket to a UTF-8 string produces an ASCII-8BIT string. On Ruby 3.2 the concatenation produces a UTF-8 string. Here's a program demonstrating the behavior of the SSL socket: ```ruby # ssltest.rb require 'openssl' require 'uri' url = URI(ARGV.first) path = url.path path += '?' + url.query if url.query req = "GET #{path} HTTP/1.1\r\nHost: #{url.hostname}\r\nAccept: */*\r\n\r\n" sock = OpenSSL::SSL::SSLSocket.open(url.hostname, url.port || HTTPS.default_https_port) sock.connect sock.write(req) sleep(1) loop do sleep(0.1) b = ''.b r = sock.read_nonblock(1024 * 16, b, exception: false) break unless String === r p [r.bytesize, r.encoding.to_s, r.ascii_only?] end ``` Ruby 3.1: ``` $ ruby ssltest.rb https://www.gnu.org/software/gzip/manual/gzip.pdf [475, "ASCII-8BIT", true] [16384, "ASCII-8BIT", false] # <== always false (except HTTP header): CORRECT [16384, "ASCII-8BIT", false] ... [13927, "ASCII-8BIT", false] ``` Ruby 3.2: ``` $ ruby ssltest.rb https://www.gnu.org/software/gzip/manual/gzip.pdf [475, "ASCII-8BIT", true] [16384, "ASCII-8BIT", true] # <== always true: WRONG [16384, "ASCII-8BIT", true] ... [13927, "ASCII-8BIT", true] ``` -- https://bugs.ruby-lang.org/
Issue #19468 has been updated by romuloceccon (Rômulo Ceccon). Another way to reproduce the problem: ``` $ ruby require "zlib" p Zlib::Inflate.new.inflate(Zlib.deflate("\u3042".b)).ascii_only? p Zlib::Inflate.new.inflate(Zlib.deflate("\u3042".b), buffer: "".b).ascii_only? ^D false # <== correct true # <== wrong ``` Bisecting points to https://github.com/ruby/ruby/commit/b0b9f7201a as the offending commit. This patch fixes the issue with both zlib and openssl: ```patch diff --git a/string.c b/string.c index 6773ed2d86..95f023232a 100644 --- a/string.c +++ b/string.c @@ -2460,6 +2460,7 @@ rb_str_modify_expand(VALUE str, long expand) else if (expand > 0) { RESIZE_CAPA_TERM(str, len + expand, termlen); } + ENC_CODERANGE_CLEAR(str); } /* As rb_str_modify(), but don't clear coderange */ ``` I could fix zlib's `rb_inflate_inflate` (https://github.com/ruby/ruby/blob/65ab2c1ef2/ext/zlib/zlib.c#L2167) and openssl's `ossl_ssl_read_internal` (https://github.com/ruby/ruby/blob/65ab2c1ef2/ext/openssl/ossl_ssl.c#L1929) instead, but I don't fully understand the reasoning of the change. ---------------------------------------- Bug #19468: Ruby 3.2: net/http sets UTF-8 encoding for binary responses https://bugs.ruby-lang.org/issues/19468#change-102128 * Author: romuloceccon (Rômulo Ceccon) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.1 (2023-02-22 revision 65ab2c1ef2) [x86_64-linux] * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- net/http on Ruby 3.2 has changed the encoding of binary responses from SSL connected hosts (non-SSL connections are not affected): ``` ruby # req.rb require 'openssl' require 'net/http' puts "openssl ext: #{OpenSSL::VERSION}" puts "openssl lib: #{OpenSSL::OPENSSL_VERSION}" puts "net-protocol: #{Net::Protocol::VERSION}" puts "net-http: #{Net::HTTP::VERSION}" puts Net::HTTP.get(URI(ARGV.first)).encoding ``` Ruby 3.1 (with updated net-protocol and net-http libs): ``` $ ruby -v ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [x86_64-linux] $ ruby req.rb https://www.gnu.org/software/gzip/manual/gzip.pdf openssl ext: 3.0.0 openssl lib: OpenSSL 1.1.1n 15 Mar 2022 net-protocol: 0.2.1 net-http: 0.3.2 ASCII-8BIT # <== CORRECT ``` Ruby 3.2 (latest git revision): ``` $ ruby -v ruby 3.2.1 (2023-02-22 revision 65ab2c1ef2) [x86_64-linux] $ ruby req.rb https://www.gnu.org/software/gzip/manual/gzip.pdf openssl ext: 3.1.0 openssl lib: OpenSSL 1.1.1n 15 Mar 2022 net-protocol: 0.2.1 net-http: 0.3.2 UTF-8 # <== WRONG ``` I've tracked the problem down to the SSL socket call at https://github.com/ruby/ruby/blob/9557c8edf2dcf18fdece066c596a71696b2f2b30/l.... The string returned has the encoding set to `ASCII-8BIT`, but `#ascii_only?` also always reports true, even when there are non-ascii bytes. This seems to be a bug, and is the probably cause of the change in behavior in net/http. On Ruby 3.1 concatenating the result of reading the SSL socket to a UTF-8 string produces an ASCII-8BIT string. On Ruby 3.2 the concatenation produces a UTF-8 string. Here's a program demonstrating the behavior of the SSL socket: ```ruby # ssltest.rb require 'openssl' require 'uri' url = URI(ARGV.first) path = url.path path += '?' + url.query if url.query req = "GET #{path} HTTP/1.1\r\nHost: #{url.hostname}\r\nAccept: */*\r\n\r\n" sock = OpenSSL::SSL::SSLSocket.open(url.hostname, url.port || HTTPS.default_https_port) sock.connect sock.write(req) sleep(1) loop do sleep(0.1) b = ''.b r = sock.read_nonblock(1024 * 16, b, exception: false) break unless String === r p [r.bytesize, r.encoding.to_s, r.ascii_only?] end ``` Ruby 3.1: ``` $ ruby ssltest.rb https://www.gnu.org/software/gzip/manual/gzip.pdf [475, "ASCII-8BIT", true] [16384, "ASCII-8BIT", false] # <== always false (except HTTP header): CORRECT [16384, "ASCII-8BIT", false] ... [13927, "ASCII-8BIT", false] ``` Ruby 3.2: ``` $ ruby ssltest.rb https://www.gnu.org/software/gzip/manual/gzip.pdf [475, "ASCII-8BIT", true] [16384, "ASCII-8BIT", true] # <== always true: WRONG [16384, "ASCII-8BIT", true] ... [13927, "ASCII-8BIT", true] ``` -- https://bugs.ruby-lang.org/
Issue #19468 has been updated by byroot (Jean Boussier).
This patch fixes the issue with both zlib and openssl:
Interesting, that you for the bug report. One problem though is that the point of https://github.com/ruby/ruby/commit/b0b9f7201a, as to not clear the coderange as often when it wasn't required. Expanding the string capacity shouldn't change the coderange. So the missing `CODERANGE_CLEAR` is likely in whatever function calls this one. I'll try to see if I can track it down. ---------------------------------------- Bug #19468: Ruby 3.2: net/http sets UTF-8 encoding for binary responses https://bugs.ruby-lang.org/issues/19468#change-102129 * Author: romuloceccon (Rômulo Ceccon) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.1 (2023-02-22 revision 65ab2c1ef2) [x86_64-linux] * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- net/http on Ruby 3.2 has changed the encoding of binary responses from SSL connected hosts (non-SSL connections are not affected): ``` ruby # req.rb require 'openssl' require 'net/http' puts "openssl ext: #{OpenSSL::VERSION}" puts "openssl lib: #{OpenSSL::OPENSSL_VERSION}" puts "net-protocol: #{Net::Protocol::VERSION}" puts "net-http: #{Net::HTTP::VERSION}" puts Net::HTTP.get(URI(ARGV.first)).encoding ``` Ruby 3.1 (with updated net-protocol and net-http libs): ``` $ ruby -v ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [x86_64-linux] $ ruby req.rb https://www.gnu.org/software/gzip/manual/gzip.pdf openssl ext: 3.0.0 openssl lib: OpenSSL 1.1.1n 15 Mar 2022 net-protocol: 0.2.1 net-http: 0.3.2 ASCII-8BIT # <== CORRECT ``` Ruby 3.2 (latest git revision): ``` $ ruby -v ruby 3.2.1 (2023-02-22 revision 65ab2c1ef2) [x86_64-linux] $ ruby req.rb https://www.gnu.org/software/gzip/manual/gzip.pdf openssl ext: 3.1.0 openssl lib: OpenSSL 1.1.1n 15 Mar 2022 net-protocol: 0.2.1 net-http: 0.3.2 UTF-8 # <== WRONG ``` I've tracked the problem down to the SSL socket call at https://github.com/ruby/ruby/blob/9557c8edf2dcf18fdece066c596a71696b2f2b30/l.... The string returned has the encoding set to `ASCII-8BIT`, but `#ascii_only?` also always reports true, even when there are non-ascii bytes. This seems to be a bug, and is the probably cause of the change in behavior in net/http. On Ruby 3.1 concatenating the result of reading the SSL socket to a UTF-8 string produces an ASCII-8BIT string. On Ruby 3.2 the concatenation produces a UTF-8 string. Here's a program demonstrating the behavior of the SSL socket: ```ruby # ssltest.rb require 'openssl' require 'uri' url = URI(ARGV.first) path = url.path path += '?' + url.query if url.query req = "GET #{path} HTTP/1.1\r\nHost: #{url.hostname}\r\nAccept: */*\r\n\r\n" sock = OpenSSL::SSL::SSLSocket.open(url.hostname, url.port || HTTPS.default_https_port) sock.connect sock.write(req) sleep(1) loop do sleep(0.1) b = ''.b r = sock.read_nonblock(1024 * 16, b, exception: false) break unless String === r p [r.bytesize, r.encoding.to_s, r.ascii_only?] end ``` Ruby 3.1: ``` $ ruby ssltest.rb https://www.gnu.org/software/gzip/manual/gzip.pdf [475, "ASCII-8BIT", true] [16384, "ASCII-8BIT", false] # <== always false (except HTTP header): CORRECT [16384, "ASCII-8BIT", false] ... [13927, "ASCII-8BIT", false] ``` Ruby 3.2: ``` $ ruby ssltest.rb https://www.gnu.org/software/gzip/manual/gzip.pdf [475, "ASCII-8BIT", true] [16384, "ASCII-8BIT", true] # <== always true: WRONG [16384, "ASCII-8BIT", true] ... [13927, "ASCII-8BIT", true] ``` -- https://bugs.ruby-lang.org/
Issue #19468 has been updated by byroot (Jean Boussier). Actually, the assumption in https://github.com/ruby/ruby/commit/b0b9f7201a might have been incorrect. The function is called `rb_str_modify_expand` and `rb_str_modify` does clear the coderange, so it is somewhat legitimate for extension to assume it include what `rb_str_modify` does. SO your patch seem correct to me. ---------------------------------------- Bug #19468: Ruby 3.2: net/http sets UTF-8 encoding for binary responses https://bugs.ruby-lang.org/issues/19468#change-102130 * Author: romuloceccon (Rômulo Ceccon) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.1 (2023-02-22 revision 65ab2c1ef2) [x86_64-linux] * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- net/http on Ruby 3.2 has changed the encoding of binary responses from SSL connected hosts (non-SSL connections are not affected): ``` ruby # req.rb require 'openssl' require 'net/http' puts "openssl ext: #{OpenSSL::VERSION}" puts "openssl lib: #{OpenSSL::OPENSSL_VERSION}" puts "net-protocol: #{Net::Protocol::VERSION}" puts "net-http: #{Net::HTTP::VERSION}" puts Net::HTTP.get(URI(ARGV.first)).encoding ``` Ruby 3.1 (with updated net-protocol and net-http libs): ``` $ ruby -v ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [x86_64-linux] $ ruby req.rb https://www.gnu.org/software/gzip/manual/gzip.pdf openssl ext: 3.0.0 openssl lib: OpenSSL 1.1.1n 15 Mar 2022 net-protocol: 0.2.1 net-http: 0.3.2 ASCII-8BIT # <== CORRECT ``` Ruby 3.2 (latest git revision): ``` $ ruby -v ruby 3.2.1 (2023-02-22 revision 65ab2c1ef2) [x86_64-linux] $ ruby req.rb https://www.gnu.org/software/gzip/manual/gzip.pdf openssl ext: 3.1.0 openssl lib: OpenSSL 1.1.1n 15 Mar 2022 net-protocol: 0.2.1 net-http: 0.3.2 UTF-8 # <== WRONG ``` I've tracked the problem down to the SSL socket call at https://github.com/ruby/ruby/blob/9557c8edf2dcf18fdece066c596a71696b2f2b30/l.... The string returned has the encoding set to `ASCII-8BIT`, but `#ascii_only?` also always reports true, even when there are non-ascii bytes. This seems to be a bug, and is the probably cause of the change in behavior in net/http. On Ruby 3.1 concatenating the result of reading the SSL socket to a UTF-8 string produces an ASCII-8BIT string. On Ruby 3.2 the concatenation produces a UTF-8 string. Here's a program demonstrating the behavior of the SSL socket: ```ruby # ssltest.rb require 'openssl' require 'uri' url = URI(ARGV.first) path = url.path path += '?' + url.query if url.query req = "GET #{path} HTTP/1.1\r\nHost: #{url.hostname}\r\nAccept: */*\r\n\r\n" sock = OpenSSL::SSL::SSLSocket.open(url.hostname, url.port || HTTPS.default_https_port) sock.connect sock.write(req) sleep(1) loop do sleep(0.1) b = ''.b r = sock.read_nonblock(1024 * 16, b, exception: false) break unless String === r p [r.bytesize, r.encoding.to_s, r.ascii_only?] end ``` Ruby 3.1: ``` $ ruby ssltest.rb https://www.gnu.org/software/gzip/manual/gzip.pdf [475, "ASCII-8BIT", true] [16384, "ASCII-8BIT", false] # <== always false (except HTTP header): CORRECT [16384, "ASCII-8BIT", false] ... [13927, "ASCII-8BIT", false] ``` Ruby 3.2: ``` $ ruby ssltest.rb https://www.gnu.org/software/gzip/manual/gzip.pdf [475, "ASCII-8BIT", true] [16384, "ASCII-8BIT", true] # <== always true: WRONG [16384, "ASCII-8BIT", true] ... [13927, "ASCII-8BIT", true] ``` -- https://bugs.ruby-lang.org/
Issue #19468 has been updated by naruse (Yui NARUSE). Backport changed from 2.7: DONTNEED, 3.0: DONTNEED, 3.1: DONTNEED, 3.2: REQUIRED to 2.7: DONTNEED, 3.0: DONTNEED, 3.1: DONTNEED, 3.2: DONE ruby_3_2 b309c246ee70926d593d3857e1625202e2d0f67b merged revision(s) d78ae78fd76e556e281a743c75bea4c0bb81ed8c. ---------------------------------------- Bug #19468: Ruby 3.2: net/http sets UTF-8 encoding for binary responses https://bugs.ruby-lang.org/issues/19468#change-102446 * Author: romuloceccon (Rômulo Ceccon) * Status: Closed * Priority: Normal * ruby -v: ruby 3.2.1 (2023-02-22 revision 65ab2c1ef2) [x86_64-linux] * Backport: 2.7: DONTNEED, 3.0: DONTNEED, 3.1: DONTNEED, 3.2: DONE ---------------------------------------- net/http on Ruby 3.2 has changed the encoding of binary responses from SSL connected hosts (non-SSL connections are not affected): ``` ruby # req.rb require 'openssl' require 'net/http' puts "openssl ext: #{OpenSSL::VERSION}" puts "openssl lib: #{OpenSSL::OPENSSL_VERSION}" puts "net-protocol: #{Net::Protocol::VERSION}" puts "net-http: #{Net::HTTP::VERSION}" puts Net::HTTP.get(URI(ARGV.first)).encoding ``` Ruby 3.1 (with updated net-protocol and net-http libs): ``` $ ruby -v ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [x86_64-linux] $ ruby req.rb https://www.gnu.org/software/gzip/manual/gzip.pdf openssl ext: 3.0.0 openssl lib: OpenSSL 1.1.1n 15 Mar 2022 net-protocol: 0.2.1 net-http: 0.3.2 ASCII-8BIT # <== CORRECT ``` Ruby 3.2 (latest git revision): ``` $ ruby -v ruby 3.2.1 (2023-02-22 revision 65ab2c1ef2) [x86_64-linux] $ ruby req.rb https://www.gnu.org/software/gzip/manual/gzip.pdf openssl ext: 3.1.0 openssl lib: OpenSSL 1.1.1n 15 Mar 2022 net-protocol: 0.2.1 net-http: 0.3.2 UTF-8 # <== WRONG ``` I've tracked the problem down to the SSL socket call at https://github.com/ruby/ruby/blob/9557c8edf2dcf18fdece066c596a71696b2f2b30/l.... The string returned has the encoding set to `ASCII-8BIT`, but `#ascii_only?` also always reports true, even when there are non-ascii bytes. This seems to be a bug, and is the probably cause of the change in behavior in net/http. On Ruby 3.1 concatenating the result of reading the SSL socket to a UTF-8 string produces an ASCII-8BIT string. On Ruby 3.2 the concatenation produces a UTF-8 string. Here's a program demonstrating the behavior of the SSL socket: ```ruby # ssltest.rb require 'openssl' require 'uri' url = URI(ARGV.first) path = url.path path += '?' + url.query if url.query req = "GET #{path} HTTP/1.1\r\nHost: #{url.hostname}\r\nAccept: */*\r\n\r\n" sock = OpenSSL::SSL::SSLSocket.open(url.hostname, url.port || HTTPS.default_https_port) sock.connect sock.write(req) sleep(1) loop do sleep(0.1) b = ''.b r = sock.read_nonblock(1024 * 16, b, exception: false) break unless String === r p [r.bytesize, r.encoding.to_s, r.ascii_only?] end ``` Ruby 3.1: ``` $ ruby ssltest.rb https://www.gnu.org/software/gzip/manual/gzip.pdf [475, "ASCII-8BIT", true] [16384, "ASCII-8BIT", false] # <== always false (except HTTP header): CORRECT [16384, "ASCII-8BIT", false] ... [13927, "ASCII-8BIT", false] ``` Ruby 3.2: ``` $ ruby ssltest.rb https://www.gnu.org/software/gzip/manual/gzip.pdf [475, "ASCII-8BIT", true] [16384, "ASCII-8BIT", true] # <== always true: WRONG [16384, "ASCII-8BIT", true] ... [13927, "ASCII-8BIT", true] ``` -- https://bugs.ruby-lang.org/
participants (3)
-
byroot (Jean Boussier) -
naruse (Yui NARUSE)
-
romuloceccon