February 2024 - ruby-core - ml.ruby-lang.org

[ruby-core:116790] [Ruby master Bug#20270] Options with `--parser=prism`
by nobu (Nobuyoshi Nakada) 11 Jun '24

11 Jun '24

Issue #20270 has been reported by nobu (Nobuyoshi Nakada). ---------------------------------------- Bug #20270: Options with `--parser=prism` https://bugs.ruby-lang.org/issues/20270 * Author: nobu (Nobuyoshi Nakada) * Status: Open * Priority: Normal * Backport: 3.0: DONTNEED, 3.1: DONTNEED, 3.2: DONTNEED, 3.3: REQUIRED ---------------------------------------- ### `--dump` option Currently `parsetree` and `prism_parsetree` bits are separated, but it seems meaningless as far as `--parser` option selects only one parser. Why doesn't simply `--dump=parsetree` dump AST by parse.y, or PRISM AST if `--parser=prism` is given? Actually, `ruby --dump=parsetree --parser=prism` just segfaults, because these bits and `--parser` option are separately handled. ### streaming code from stdin ```sh-session $ echo end | ruby --parser=prism - ruby: warning: The compiler based on the Prism parser is currently experimental and compatibility with the compiler based on parse.y is not yet complete. Please report any issues you find on the `ruby/prism` issue tracker. -: warning: Prism support for streaming code from stdin is not currently supported ``` Nothing reported is fine, since it is not currently supported. I was a bit curious that it looks like trying something after the warning. ```C if (strcmp(opt->script, "-") == 0) { int xflag = opt->xflag; VALUE rb_source = open_load_file(opt->script_name, &xflag); opt->xflag = xflag != 0; rb_warn("Prism support for streaming code from stdin is not currently supported"); error = pm_parse_string(&result, rb_source, opt->script_name); } ``` Note that `open_load_file` returns `rb_stdin` when `-` is given. This object is a `RFile`, not a `RString`, and `RSTRING_PTR(rb_source)` accesses after the object boundary. It should just bail out, not try obviously wrong thing. ### Fix https://github.com/nobu/ruby/tree/options-refactor -- https://bugs.ruby-lang.org/

2 3

[ruby-core:116039] [Ruby master Bug#20154] aarch64: configure overrides `-mbranch-protection` if it was set in CFLAGS via environment
by jprokop (Jarek Prokop) 09 Jun '24

09 Jun '24

Issue #20154 has been reported by jprokop (Jarek Prokop). ---------------------------------------- Bug #20154: aarch64: configure overrides `-mbranch-protection` if it was set in CFLAGS via environment https://bugs.ruby-lang.org/issues/20154 * Author: jprokop (Jarek Prokop) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- Recently a GH PR was merged <https://github.com/ruby/ruby/pull/9306> For PAC/BTI support on ARM CPUs for Coroutine.S. Without proper compilation support in configure.ac it segfaults Ruby with fibers on CPUs where PAC is supported: https://bugs.ruby-lang.org/issues/20085 At the time of writing, configure.ac appends the first option from a list for flag `-mbranch-protection` that successfully compiles a program <https://github.com/ruby/ruby/blob/master/configure.ac#L829>, to XCFLAGS and now also ASFLAGS to fix issue 20085 for Ruby master. This is suboptimal for Fedora as we set -mbranch-protection=standard by default in C{,XX}FLAGS: ``` CFLAGS='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Werror=implicit-function-declaration -Werror=implicit-int -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -mbranch-protection=standard -fasynchronous-unwind-tables -fstack-clash-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer ' export CFLAGS CXXFLAGS='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -mbranch-protection=standard -fasynchronous-unwind-tables -fstack-clash-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer' export CXXFLAGS ``` And the appended flag overrides distribution's compilation configuration, which in this case ends up omitting BTI instructions and only using PAC. Would it make sense to check if such flags exist and not overwrite them if they do? Serious proposals: 1. Simplest fix that does not overwrite what is set in the distribution and results in higher security is simply prepending the list of options with `-mbranch-protection=standard`, it should cause no problems on ARMv8 CPUs and forward, BTI similarly to PAC instructions result into NOP, it is only extending the capability. See attached 0001-aarch64-Check-mbranch-protection-standard-first-to-u.patch 2. Other fix that sounds more sane IMO and dodges this kind of guessing where are all the correct places for the flag is what another Fedora contributor Florian Weimer suggested: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org… "The reliable way to do this would be to compile a C file and check whether that enables __ARM_FEATURE_PAC_DEFAULT, and if that's the case, define a *different* macro for use in the assembler implementation. This way, you don't need to care about the exact name of the option." IOW instead of using __ARM_FEATURE_* directly in that code, define a macro in the style of "USE_PAC" with value of the feature if it is defined, I think that way we shouldn't need to append ASFLAGS anymore. However it's also important to catch the value of those macros as their values have meaning, I have an idea how to do that but I'd get on that monday earliest. ---Files-------------------------------- 0001-aarch64-Check-mbranch-protection-standard-first-to-u.patch (1004 Bytes) -- https://bugs.ruby-lang.org/

4 10

[ruby-core:116460] [Ruby master Bug#20218] aset/masgn/op_asgn with keyword arguments
by jeremyevans0 (Jeremy Evans) 07 Jun '24

07 Jun '24

Issue #20218 has been reported by jeremyevans0 (Jeremy Evans). ---------------------------------------- Bug #20218: aset/masgn/op_asgn with keyword arguments https://bugs.ruby-lang.org/issues/20218 * Author: jeremyevans0 (Jeremy Evans) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- I found that use of keyword arguments in multiple assignment is broken in 3.3 and master: ```ruby h = {a: 1} o = [] def o.[]=(*args, **kw) replace([args, kw]) end # This segfaults as RHS argument is not a hash o[1, a: 1], _ = [1, 2] # This passes the RHS argument as keywords to the method, treating keyword splat as positional argument o[1, **h], _ = [{b: 3}, 2] o # => [[1, {:a=>1}], {:b=>3}] ``` Before 3.3, keyword arguments were treated as positional arguments. This is similar to #19918, but for keyword arguments instead of block arguments. @matz indicated he wanted to prohibit block arguments in aset/masgn and presumably also op_asgn (making them SyntaxErrors). Can we also prohibit keyword arguments in aset/masgn/op_asgn? Note that aset treats keyword arguments as regular arguments: ```ruby o[1, a: 1] = 2 o # => [[1, {:a=>1}, 2], {}] o[1, **h] = {b: 3} o # => [[1, {:a=>2}, {:b=>3}], {}] ``` While op_asgn treats keyword arguments as keywords: ```ruby h = {a: 2} o = [] def o.[](*args, **kw) concat([:[], args, kw]) x = Object.new def x.+(v) [:x, v] end x end def o.[]=(*args, **kw) concat([:[]=, args, kw]) end o[1, a: 1] += 2 o # => [:[], [1], {:a=>1}, :[]=, [1, [:x, 2]], {:a=>1}] o.clear o[1, **h] += {b: 3} o # => [:[], [1], {:a=>2}, :[]=, [1, [:x, {:b=>3}]], {:a=>2}] ``` -- https://bugs.ruby-lang.org/

7 17

[ruby-core:116117] [Ruby master Misc#20170] Drop support for GCC < 11
by kddnewton (Kevin Newton) 07 Jun '24

07 Jun '24

Issue #20170 has been reported by kddnewton (Kevin Newton). ---------------------------------------- Misc #20170: Drop support for GCC < 11 https://bugs.ruby-lang.org/issues/20170 * Author: kddnewton (Kevin Newton) * Status: Open * Priority: Normal ---------------------------------------- Right now, CI compiles everything from GCC 7+. However, GCC 7-10 are all end-of-life and no longer supported. We should drop support for the end-of-life compilers. -- https://bugs.ruby-lang.org/

4 5

[ruby-core:115595] [Ruby master Bug#20043] `defined?` checks for method existence but only sometimes
by tenderlovemaking (Aaron Patterson) 07 Jun '24

07 Jun '24

Issue #20043 has been reported by tenderlovemaking (Aaron Patterson). ---------------------------------------- Bug #20043: `defined?` checks for method existence but only sometimes https://bugs.ruby-lang.org/issues/20043 * Author: tenderlovemaking (Aaron Patterson) * Status: Open * Priority: Normal * ruby -v: ruby 3.3.0dev (2023-12-05T21:25:34Z master 56eccb350b) [arm64-darwin23] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- When an expression is passed to `defined?`, it will _sometimes_ check if a method in a sub-expression is defined and sometimes it won't. For example: ``` $ ./miniruby -e'p defined?(a)' nil $ ./miniruby -e'p defined?([a])' nil ``` In the above case, Ruby will check whether or not the method `a` is defined, and it returns `nil`. However, if you use a splat, it will not check: ``` $ ./miniruby -e'p defined?([*a])' "expression" ``` The same thing seems to happen with method parameters: ``` $ ./miniruby -e'p defined?(itself)' "method" $ ./miniruby -e'p defined?(itself(a))' nil $ ./miniruby -e'p defined?(itself(*a))' "method" ``` Oddly, `defined?` will check contents of arrays, but _won't_ check contents of hashes: ``` $ ./miniruby -e'p defined?([[[[a]]]])' nil $ ./miniruby -e'p defined?({ a => a })' "expression" ``` I think all of the cases that refer to `a` should check whether or not `a` is defined regardless of splats or hashes. -- https://bugs.ruby-lang.org/

7 9

[ruby-core:115334] [Ruby master Feature#19998] Emit deprecation warnings when the old (non-Typed) Data_XXX API is used
by byroot (Jean Boussier) 06 Jun '24

06 Jun '24

Issue #19998 has been reported by byroot (Jean Boussier). ---------------------------------------- Feature #19998: Emit deprecation warnings when the old (non-Typed) Data_XXX API is used https://bugs.ruby-lang.org/issues/19998 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal ---------------------------------------- The replacement API was introduced in Ruby 1.9.2 (2010) [Feature #3064], and the old untyped data API was marked a deprecated in the documentation as of Ruby 2.3.0 (2015): https://github.com/ruby/ruby/commit/98544c372d948717de22afc86d162e411f1fb5f1 However, as of today, no deprecation warning of any sort is emitted when the API is used. Ultimately removing the old untyped API would allow to easily reclaim 8B on every `T_DATA` which could be used by the newly introduced embedded TypedData objects https://github.com/ruby/ruby/pull/7440 Was there any discussions about removing the old API at some point? Could we emit a warning when the API is used? Either verbose or non-verbose, to at least surface the issue to gem owners? This would help drive the effort to fix outdated gems, allowing to eventually remove this old API in the future. Looking at our big applications dependencies, there seem to be a small minority of gems still using the old API: - https://rubygems.org/gems/unf_ext - https://rubygems.org/gems/rotoscope - https://rubygems.org/gems/re2 - https://rubygems.org/gems/mysql2 Migrating them over doesn't seem too hard. -- https://bugs.ruby-lang.org/

6 11

[ruby-core:114038] [Ruby master Bug#19749] Confirm correct behaviour when attaching private method with `#define_method`
by itarato (Peter Arato) 06 Jun '24

06 Jun '24

Issue #19749 has been reported by itarato (Peter Arato). ---------------------------------------- Bug #19749: Confirm correct behaviour when attaching private method with `#define_method` https://bugs.ruby-lang.org/issues/19749 * Author: itarato (Peter Arato) * Status: Open * Priority: Normal * ruby -v: 3.3.0 * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- This issue is a special case of https://bugs.ruby-lang.org/issues/19745: Should dynamically added private methods via `.singleton_class.send(:define_method,...` be accessible publicly? See the following example? ```ruby private def bar; end foo = Object.new foo.singleton_class.send(:define_method, :bar, method(:bar)) foo.bar # No error. ``` The script above runs fine on latest Ruby 3.3. Is this correct to ignore the fact that the added method (method(:bar)) is private? This came up during a TruffleRuby investigation (https://github.com/oracle/truffleruby/issues/3134) where the result for the same script is: `private method 'bar' called for #<Object:0xc8> (NoMethodError)` -- https://bugs.ruby-lang.org/

5 14

[ruby-core:112538] [Ruby master Feature#19458] Expose HEREDOC identifier
by joelhawksley (Joel Hawksley) 04 Jun '24

04 Jun '24

Issue #19458 has been reported by joelhawksley (Joel Hawksley). ---------------------------------------- Feature #19458: Expose HEREDOC identifier https://bugs.ruby-lang.org/issues/19458 * Author: joelhawksley (Joel Hawksley) * Status: Open * Priority: Normal ---------------------------------------- I’d like to have access to the HEREDOC identifier. In the ViewComponent framework I help maintain, we added a method to declare a template as such: ```ruby class Component erb_template <<~ERB <h1>Hello, <%= @name %>!</h1> ERB end ``` I'd prefer to be able to write: ```ruby class Component template <<~ERB <h1>Hello, <%= @name %>!</h1> ERB end ``` And be able to see that the argument passed to `.template` was from a HEREDOC with an `ERB` identifier, which would allow me to use the correct template handler to compile the template. I could see this being implemented: 1) As a new property of String, such as `identifier` or `heredoc_identifier`. 2) By having HEREDOCs return a subclass of String that includes an `identifier` property. I'd be happy to work on implementing this change. -- https://bugs.ruby-lang.org/

9 13

[ruby-core:115923] [Ruby master Bug#20094] Inline while loop behavior changed unexpectedly in 3.3.0
by sisyphus_cg (Sisyphus CG) 30 May '24

30 May '24

Issue #20094 has been reported by sisyphus_cg (Sisyphus CG). ---------------------------------------- Bug #20094: Inline while loop behavior changed unexpectedly in 3.3.0 https://bugs.ruby-lang.org/issues/20094 * Author: sisyphus_cg (Sisyphus CG) * Status: Open * Priority: Normal * ruby -v: 3.3.0 * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- The behavior of the inline while loop has changed in 3.3.0. This unexpectedly broke my code and I couldn't find anything in the changelog about it so reporting it as a bug. In ruby <= 3.2: ``` ruby (p 1) while false # nothing (p 1; p 2) while false # nothing ``` In ruby 3.3: ``` ruby (p 1) while false # 1 (p 1; p 2) while false # 1 2 ``` Essentially, if the left hand side looks like a statement, now it is treated as a do-while loop. In ruby 3.2 and less this only happened with explicit use of `begin` and `end`. -- https://bugs.ruby-lang.org/

9 11

[ruby-core:116901] [Ruby master Bug#20289] Bug in Zlib::GzipReader#eof? breaks reading certain sizes of gzipped files.
by martinemde (Martin Emde) 29 May '24

29 May '24

Issue #20289 has been reported by martinemde (Martin Emde). ---------------------------------------- Bug #20289: Bug in Zlib::GzipReader#eof? breaks reading certain sizes of gzipped files. https://bugs.ruby-lang.org/issues/20289 * Author: martinemde (Martin Emde) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- Hello, A bug in the implementation of Zlib::GzipReader#eof? makes it very difficult to read certain rubygems using readpartial without receiving an EOFError. The bug is caused when the chunk size read from the gzip leaves only empty gzip "overhead" bytes at the end of the unread portion. The result is that a simple `readpartial until eof?` fails on some real world gems. ### Bug Explanation Imagine a gzip file, it has compressed data and some bytes that indicate the start and end of a chunk and then end of a file. ``` gzip file = "[... 2048 chunk of gzip data ...][ \0 bytes, chucksum info, with no new readable bytes ]" ``` When reading this file, zlib.c will pull a chunk of data according to READ_SIZE and return that data uncompressed. With exactly the right size of data (within exactly the range of READ-SIZE*n - 24 to READ_SIZE*n - 15 for any multiple of READ_SIZE) all of the file data is read and returned on the first read. The bug happens when there are bytes remaining on the gzip file that haven't been read, but contain no new data that can be returned to the buffer. Before reading the final empty chunk, #eof? returns false. After reading the final chunk, EOFError is raised because no bytes were returned into the buffer. What should happen is similar to how a Socket eof? is checked. On a socket, #eof? is not "passive" but actively reads ahead, filling the buffer. If that read-ahead fails to find new data, then the EOF is reached and eof? returns true. In zlib, #eof? does not read ahead to check if the end has been reached, causing this bug. ### Solution Samuel Giddins and I have both submitted alternate, but very similar solutions for this bug. [Samuel's PR](https://github.com/ruby/zlib/pull/73) and [My PR](https://github.com/ruby/zlib/pull/72). We'd leave it to ruby core to decide which is the best solution. Unfortunately, these bug fix PRs have been waiting for more than 2 months now with no movement on fixing this bug. The [original issue](https://github.com/ruby/zlib/issues/56) that I opened to point out this bug was submitted Aug 16, 2023. We've been working to find a solution for this bug for more than half a year. This bug was discovered because rubygems wanted to improve the efficiency of reading a gem by using readpartial to conserve memory. Sam submitted an additional PR that vastly improves the performance of the zlib gem by managing buffers better. We've already proven that this dramatically decreases memory usage when parsing gems. The [performance PR by Samuel](https://github.com/ruby/zlib/pull/61) has been open since September 13th, 2023. ### Plea We've been trying for months to get this solved. Opening a ruby-lang bug is my next attempt after communicating through our contacts at Ruby core has gone nowhere. Please merge and release these fixes, or tell us what is wrong with them so we can fix them so they can be merged. It will improve rubygems for everyone. -- https://bugs.ruby-lang.org/

3 2