August 2024 - ruby-core - ml.ruby-lang.org

[ruby-core:118923] [Ruby master Bug#20690] URI.encode_www_form_component method escapes tilde when it's not supposed to
by y4m4p (Masahiro Yamashita) 06 Dec '24

06 Dec '24

Issue #20690 has been reported by y4m4p (Masahiro Yamashita). ---------------------------------------- Bug #20690: URI.encode_www_form_component method escapes tilde when it's not supposed to https://bugs.ruby-lang.org/issues/20690 * Author: y4m4p (Masahiro Yamashita) * Status: Open * ruby -v: ruby 3.2.2 (2023-03-30 revision e51014f9c0) [arm64-darwin21] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- ## Problem As the title says, `Uri.encode_www_form_component` is escaping tilde (`~`) where it should not according to the RFC3986. Ref: https://datatracker.ietf.org/doc/html/rfc3986#section-2.3 There was a previous issue with the same problem, https://bugs.ruby-lang.org/issues/6696 which has been resolved and merged. --- Since `URI.escape` is now obsolete and gone, the preferred way to escape URI characters are to choose from the following methods. - `URI.encode_www_form_component` (or, `URI.encode_www_form` which uses this) - `CGI.escape` - `ERB::Util.#url_encode` - `WEBrick::HTTPUtils.#escape_form` - `WEBrick::HTTPUtils.#escape` This issue does not occur when using any of the methods except `URI.encode_www_form_component`. These preferred options are not compatible with each other, making it a poor experience for Ruby users. ### Minimum replication ``` ruby # URI irb(main):002:0> require 'uri' => true irb(main):004:0> URI.encode_www_form_component("ruby~test") => "ruby%7Etest" # CGI irb(main):001:0> require 'cgi' => true irb(main):003:0> CGI.escape("ruby~test") => "ruby~test" # ERB irb(main):009:0> require 'erb' => true irb(main):012:0> ERB::Util.url_encode("ruby~test") => "ruby~test" # Webrick # needs webrick gem installed [3] pry(main)> require 'webrick' => true [4] pry(main)> WEBrick::HTTPUtils.escape_form("ruby~test") => "ruby~test" [5] pry(main)> WEBrick::HTTPUtils.escape("ruby~test") => "ruby~test" ``` ## Real world use-case problem I have a use-case problem with using URI when implementing a OIDC with PKCE client / server setup. When trying to generate a GET URI for Open ID Connect with PKCE, you need to provide a `code_verifier` value to the Authorization server, which actually requires you to use the tilde without escaping. Ref: https://www.oauth.com/oauth2-servers/pkce/authorization-request/ > When the native app begins the authorization request, instead of immediately launching a browser, the client first creates what is known as a “code verifier“. This is a cryptographically random string using the characters A-Z, a-z, 0-9, and the punctuation characters -._~ (hyphen, period, underscore, and tilde), between 43 and 128 characters long. The inconsistent encoding is problematic for the Server and Client both, since both ends needs to have an agreement on using tilde as an escaped character or an unescaped character. Logically speaking, `~` should be sent AS-IS and not to be escaped, since the `code_challenge` value and `code_verifier` values are strictly checked using a Hash generation function `Digest::SHA256.digest`, which generates a different Hash value depending on `~`(non escaped) or `%7E`(escaped). These parameters from user inputs are usually not meant to be tampered with for security sake, so it's best to have tilde `~` to be unescaped from the beginning. ## Related issues - https://bugs.ruby-lang.org/issues/6696 - This seems to have been resolved and merged. -- https://bugs.ruby-lang.org/

3 3

[ruby-core:111434] [Ruby master Feature#19261] `Data#members` is not important
by ko1 (Koichi Sasada) 04 Dec '24

04 Dec '24

Issue #19261 has been reported by ko1 (Koichi Sasada). ---------------------------------------- Feature #19261: `Data#members` is not important https://bugs.ruby-lang.org/issues/19261 * Author: ko1 (Koichi Sasada) * Status: Open * Priority: Normal ---------------------------------------- `Data#members` is defined but it is calculated by `self.class.members` (in other words, `#members` is a shorthand for `self.class.members`). So it is better to remove this method. ```ruby P = Data.define(:x, :y) p P.new(1, 2).members #=> [:x, :y] Group = Data.define(:name, :members) gs = Group.new('SasadaFamily', %w(ko1 yuki)) p gs.members #=> ["ko1", "yuki"] ``` -- https://bugs.ruby-lang.org/

4 5

[ruby-core:118003] [Ruby master Bug#20506] Failure compiling Ruby 3.4.0-preview1 on aarch64 on a mac and linux (Ubuntu 24.04)
by schneems (Richard Schneeman) 29 Nov '24

29 Nov '24

Issue #20506 has been reported by schneems (Richard Schneeman). ---------------------------------------- Bug #20506: Failure compiling Ruby 3.4.0-preview1 on aarch64 on a mac and linux (Ubuntu 24.04) https://bugs.ruby-lang.org/issues/20506 * Author: schneems (Richard Schneeman) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- I was unable to compile Ruby 3.4.0-preview1 on Ubuntu 24.04 via Github Actions. As the logs are not retained indefinitely here's a link to the compilation failure: https://gist.github.com/schneems/bc6bdd6fba08ff6bfdd1bce8d76c457d The relevant errors seem to be in linking yjit: ``` touch yjit/target/release/libyjit.a partial linking yjit/target/release/libyjit.a into yjit/target/release/libyjit.o linking miniruby /usr/bin/ld: yjit/target/release/libyjit.o: in function `__multc3': /cargo/registry/src/index.crates.io-6f17d22bba15001f/compiler_builtins-0.1.108/./lib/builtins/multc3.c:33:(.text.__multc3+0x30c): undefined reference to `__builtin_copysignq' /usr/bin/ld: /cargo/registry/src/index.crates.io-6f17d22bba15001f/compiler_builtins-0.1.108/./lib/builtins/multc3.c:34:(.text.__multc3+0x394): undefined reference to `__builtin_copysignq' /usr/bin/ld: /cargo/registry/src/index.crates.io-6f17d22bba15001f/compiler_builtins-0.1.108/./lib/builtins/multc3.c:42:(.text.__multc3+0x7f4): undefined reference to `__builtin_copysignq' /usr/bin/ld: /cargo/registry/src/index.crates.io-6f17d22bba15001f/compiler_builtins-0.1.108/./lib/builtins/multc3.c:43:(.text.__multc3+0x864): undefined reference to `__builtin_copysignq' /usr/bin/ld: /cargo/registry/src/index.crates.io-6f17d22bba15001f/compiler_builtins-0.1.108/./lib/builtins/multc3.c:47:(.text.__multc3+0x8c4): undefined reference to `__builtin_copysignq' /usr/bin/ld: yjit/target/release/libyjit.o:/cargo/registry/src/index.crates.io-6f17d22bba15001f/compiler_builtins-0.1.108/./lib/builtins/multc3.c:38: more undefined references to `__builtin_copysignq' follow collect2: error: ld returned 1 exit status make: *** [Makefile:298: miniruby] Error 1 /tmp/lib/build_script.rb:118:in `pipe': Command failed debugflags="-g" ./configure --disable-install-doc --prefix /tmp/d20240524-1-bdq0bj/prefix --enable-load-relative --enable-shared --enable-yjit && make -j16 && make install (RuntimeError) from /tmp/lib/build_script.rb:158:in `block in build' from /tmp/lib/build_script.rb:152:in `chdir' from /tmp/lib/build_script.rb:152:in `build' from /tmp/lib/build_script.rb:65:in `block in run_build_script' from /usr/lib/ruby/3.2.0/tmpdir.rb:94:in `mktmpdir' from /tmp/lib/build_script.rb:61:in `run_build_script' from /tmp/build.rb:8:in `<main>' ``` Here's the GHA link: https://github.com/heroku/docker-heroku-ruby-builder/actions/runs/922553814…. This is the Dockerfile for Ubuntu 24.04 https://github.com/heroku/docker-heroku-ruby-builder/blob/73bdda974811717ba…. I'm able to reproduce the failure locally: ``` $ git clone https://github.com/heroku/docker-heroku-ruby-builder $ cd docker-heroku-ruby-builder $ export DOCKER_DEFAULT_PLATFORM=linux/arm64 $ bin/activate_docker heroku-24 $ bin/build_ruby heroku-24 3.4.0-preview1 # Fails ``` Full output https://gist.github.com/schneems/cfa21847228d48e6bbe27463deae6745. I'm also seeing failures when using `ruby-install` on my mac (ARM), the error message is different so I'm not sure if it's the same problem or unrelated: https://gist.github.com/schneems/12af87e967126c7bfa319645b00015b3 -- https://bugs.ruby-lang.org/

6 11

[ruby-core:116760] [Ruby master Feature#20265] Deprecate and remove rb_newobj and rb_newobj_of
by peterzhu2118 (Peter Zhu) 28 Nov '24

28 Nov '24

Issue #20265 has been reported by peterzhu2118 (Peter Zhu). ---------------------------------------- Feature #20265: Deprecate and remove rb_newobj and rb_newobj_of https://bugs.ruby-lang.org/issues/20265 * Author: peterzhu2118 (Peter Zhu) * Status: Open * Priority: Normal ---------------------------------------- GitHub PR: https://github.com/ruby/ruby/pull/9964 I’m proposing deprecating and removing the rb_newobj and rb_newobj_of APIs because they are difficult to use, fragile to use, and requires knowledge of the internal implementation of data types in Ruby. The rb_newobj function creates a T_NONE object. T_NONE objects are tricky to deal with since T_NONE objects cannot be marked, T_NONE objects are not reclaimed by the GC, and changing the object to other types require internal knowledge about the data type. T_NONE objects are not allowed to be marked, so it cannot be GC managed. Since T_NONE objects are skipped during sweeping, it will leak Ruby heap memory if the developer never changes the object to another type. Changing a T_NONE object to another type is tricky. For example, T_STRING objects have many flags for embedded, shared, shared root, encoding, coderange, etc. Many of these flags are not public, preventing direct use by developers. Developers must understand these flags to convert a T_NONE object into a T_STRING object. While the rb_newobj_of function is easier to use than the rb_newobj function, it still requires developers to understand flags, meaning some issues of rb_newobj also apply to rb_newobj_of. Below is the usage of RB_NEWOBJ, rb_newobj, rb_newobj_of, RB_NEWOBJ_OF with vendored Ruby and comments removed. You can see that there are very few gems using these APIs and all are from over a decade ago (the most recent one is from 2011). ``` 2009-11-18 /srv/gems/bleak_house-7.2/ruby/ruby-1.8.7.patch:@@ -438,10 +438,8 @@ rb_newobj() 2023-07-01 /srv/gems/daqing_rucaptcha-3.2.2/ext/rucaptcha/target/release/build/rb-sys-6bdd5b2895b9570a/out/bindings-0.9.78-arm64-darwin22-3.2.2.rs: pub fn rb_newobj() -> VALUE; 2023-07-01 /srv/gems/daqing_rucaptcha-3.2.2/ext/rucaptcha/target/release/build/rb-sys-6bdd5b2895b9570a/out/bindings-0.9.78-arm64-darwin22-3.2.2.rs: pub fn rb_newobj_of(klass: VALUE, flags: VALUE) -> VALUE; 2010-08-06 /srv/gems/langscan-1.2/ext/langscan/ruby/compat/ripper/ripper.c: NODE *n = (NODE*)rb_newobj(); 2011-02-03 /srv/gems/memprof-0.3.10/ext/memprof.c: VALUE ret = rb_newobj(); 2011-02-03 /srv/gems/memprof-0.3.10/ext/tracers/objects.c: last_obj = orig_rb_newobj(); 2006-02-28 /srv/gems/refe-0.8.0.3/data/refe/function_source/rb/newobj:rb_newobj() 2006-02-28 /srv/gems/refe-0.8.0.3/data/refe/function_source/rb/node/newnode: NODE *n = (NODE*)rb_newobj(); 2011-08-31 /srv/gems/ruby-rpm-1.3.1/ext/rpm/dependency.c: prov = rb_newobj(); 2011-08-31 /srv/gems/ruby-rpm-1.3.1/ext/rpm/dependency.c: req = rb_newobj(); 2011-08-31 /srv/gems/ruby-rpm-1.3.1/ext/rpm/dependency.c: conf = rb_newobj(); 2011-08-31 /srv/gems/ruby-rpm-1.3.1/ext/rpm/dependency.c: obso = rb_newobj(); 2011-08-31 /srv/gems/ruby-rpm-1.3.1/ext/rpm/file.c: file = rb_newobj(); 2011-08-31 /srv/gems/ruby-rpm-1.3.1/ext/rpm/source.c: src = rb_newobj(); 2011-08-31 /srv/gems/ruby-rpm-1.3.1/ext/rpm/source.c: src = rb_newobj(); 2011-08-31 /srv/gems/ruby-rpm-1.3.1/ext/rpm/source.c: src = rb_newobj(); 2011-08-31 /srv/gems/ruby-rpm-1.3.1/ext/rpm/version.c: ver = rb_newobj(); 2011-08-31 /srv/gems/ruby-rpm-1.3.1/ext/rpm/version.c: ver = rb_newobj(); 2011-08-31 /srv/gems/ruby-rpm-1.3.1/ext/rpm/version.c: ver = rb_newobj(); 2006-11-30 /srv/gems/sydparse-1.2.0/sydparse.c: syd_scope = (struct SCOPE*)rb_newobj(); 2006-11-30 /srv/gems/sydparse-1.2.0/sydparse.y: syd_scope = (struct SCOPE*)rb_newobj(); ``` -- https://bugs.ruby-lang.org/

5 7

[ruby-core:111247] [Ruby master Feature#19191] Implicit console input transcoding is more desirable
by YO4 (Yoshinao Muramatsu) 22 Nov '24

22 Nov '24

Issue #19191 has been reported by YO4 (Yoshinao Muramatsu). ---------------------------------------- Feature #19191: Implicit console input transcoding is more desirable https://bugs.ruby-lang.org/issues/19191 * Author: YO4 (Yoshinao Muramatsu) * Status: Open * Priority: Normal ---------------------------------------- In response to Bug #18353, STDIN.internal_encoding are set and encoding is converted explcitly on Windows platform. For example, ```[STDIN.external_encoding, STDIN.internal_encoding] # => [Encoding::Windows-31J, Encoding::UTF-8]``` if STDIN is console. I feel that internal_encoding should be reserved for specific applications. And I think setting internal_encoding to STDIN is not foreseened. Today I found irb breaks STDIN encoding, like ``` >ruby -rirb -e "p [$stdin.external_encoding, $stdin.internal_encoding]; IRB.setup(''); IRB::Irb.new(); p [$stdin.external_encoding, $stdin.internal_encoding]" [#<Encoding:Windows-31J>, #<Encoding:UTF-8>] [#<Encoding:UTF-8>, nil] ``` We know input has console code page encoding. So we always can convert encoding from console code page to io_input_encoding(). ### proposal when reading from console on Windows, input encoding is enfoced to console code page and encoding conversion is implicitly applied. when ```set_encoding("UTF-8")``` implicitly converts console code page to UTF-8. when ```set_encoding("CP437", "UTF-8")``` implicitly converts console code page to UTF-8. external_encoding is ignored. binmode or binary input method is not affected by these specifications. set_encoding, etc. will continue to work as before, and this specification will affect only when encoding conversion on read (NEED_READCONV() and make_readconv()). -- https://bugs.ruby-lang.org/

2 5

[ruby-core:113489] [Ruby master Bug#19642] Remove vectored read/write from `io.c`.
by ioquatix (Samuel Williams) 20 Nov '24

20 Nov '24

Issue #19642 has been reported by ioquatix (Samuel Williams). ---------------------------------------- Bug #19642: Remove vectored read/write from `io.c`. https://bugs.ruby-lang.org/issues/19642 * Author: ioquatix (Samuel Williams) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- https://github.com/socketry/async/issues/228#issuecomment-1546789910 is a comment from the kernel developer who tells us that `writev` is always worse than `write` system call. A large amount of complexity in `io.c` comes from optional support from `writev`. So, I'd like to remove support for `writev`. I may try to measure the performance before/after. However it may not show much difference, except that the implementation in `io.c` can be much simpler. -- https://bugs.ruby-lang.org/

5 9

[ruby-core:111966] [Ruby master Feature#19366] Rename/alias Refinedment#refined_class => #refined_module
by zverok (Victor Shepelev) 19 Nov '24

19 Nov '24

Issue #19366 has been reported by zverok (Victor Shepelev). ---------------------------------------- Feature #19366: Rename/alias Refinedment#refined_class => #refined_module https://bugs.ruby-lang.org/issues/19366 * Author: zverok (Victor Shepelev) * Status: Open * Priority: Normal ---------------------------------------- In #12737, `Refinement#refined_class` is introduced. As "module" is more generic concept than "class", the name misleadingly implies that either this method doesn't returns refined _modules_, or modules can't be refined. This is obviously not true and trivially disproved: ```ruby module Refs refine Enumerable do def foo = puts 'foo' end end Refs.refinements[0].refined_class #=> Enumerable. Which is, well, not a class. # The refinement is usable, so it is not a mute concept: using Refs [1, 2, 3].foo # prints "foo" successfully ``` I believe we refer to "modules" when some feature applies to modules and classes. Unless there is some deeper consideration for the current naming (I don't see justification in #12737, but I might miss something), the method should be renamed or aliased. PS: Sorry for a huge amount of "nitpicky" issues after the version release. Typically, those concerns emerge while I am working on the [changelog](https://rubyreferences.github.io/rubychanges/), and I am usually trying to start the work at least a month before the release. Unfortunately, due to you-know-what, I was unable to do it in time this year. After the 3.2 changelog, I plan to switch to the "master changelog" approach (following the development with the "current unreleased changes" document) this year. Unless I'll be unable to due to reasons not in my control. -- https://bugs.ruby-lang.org/

4 3

[ruby-core:117240] [Ruby master Feature#20350] Return chilled string from Symbol#to_s
by Dan0042 (Daniel DeLorme) 12 Nov '24

12 Nov '24

Issue #20350 has been reported by Dan0042 (Daniel DeLorme). ---------------------------------------- Feature #20350: Return chilled string from Symbol#to_s https://bugs.ruby-lang.org/issues/20350 * Author: Dan0042 (Daniel DeLorme) * Status: Open ---------------------------------------- During Ruby 2.7 development there was an attempt to return a frozen string from Symbol#to_s (#16150#note-22) This had to be rolled back due to incompatibility, but now we have chilled strings (#20205) Symbol#to_s can safely return a chilled string, giving developers time to fix warnings before switching to a frozen string. -- https://bugs.ruby-lang.org/

6 9

[ruby-core:118976] [Ruby master Bug#20704] Windows: `-C` option does not work for multibyte path name
by nobu (Nobuyoshi Nakada) 08 Nov '24

08 Nov '24

Issue #20704 has been reported by nobu (Nobuyoshi Nakada). ---------------------------------------- Bug #20704: Windows: `-C` option does not work for multibyte path name https://bugs.ruby-lang.org/issues/20704 * Author: nobu (Nobuyoshi Nakada) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- Related to [Bug #20699]. ``` C:> \Ruby33-x64\bin\ruby.exe -C xテスト_test -ep \Ruby33-x64\bin\ruby.exe: Can't chdir to xテスト_test (fatal) ``` -- https://bugs.ruby-lang.org/

4 4

[ruby-core:116589] [Ruby master Misc#20238] Use prism for mk_builtin_loader.rb
by kddnewton (Kevin Newton) 05 Nov '24

05 Nov '24

Issue #20238 has been reported by kddnewton (Kevin Newton). ---------------------------------------- Misc #20238: Use prism for mk_builtin_loader.rb https://bugs.ruby-lang.org/issues/20238 * Author: kddnewton (Kevin Newton) * Status: Open * Priority: Normal ---------------------------------------- I would like to propose that we use prism for mk_builtin_loader.rb. Right now the Ruby syntax that you can use in builtin classes is restricted to the base Ruby version (2.7). This means you can't use a lot of the nicer syntax that Ruby has shipped in the last couple of years. If we switch to using prism to parse the builtin files instead of using ripper, then we can always use the latest version of Ruby syntax. A pull request for this is here: https://github.com/kddnewton/ruby/pull/65. The approach for the PR is taken from how RJIT bindgen works. -- https://bugs.ruby-lang.org/

6 21