January 2024 - ruby-core - ml.ruby-lang.org

[ruby-core:115196] [Ruby master Feature#19979] Allow methods to declare that they don't accept a block via `&nil`
by ufuk (Ufuk Kayserilioglu) 24 Jul '24

24 Jul '24

Issue #19979 has been reported by ufuk (Ufuk Kayserilioglu). ---------------------------------------- Feature #19979: Allow methods to declare that they don't accept a block via `&nil` https://bugs.ruby-lang.org/issues/19979 * Author: ufuk (Ufuk Kayserilioglu) * Status: Open * Priority: Normal ---------------------------------------- ## Abstract This feature proposes new syntax to allow methods to explicitly declare that they don't accept blocks, and makes passing of a block to such methods an error. ## Background In #15554, it was proposed to automatically detect methods that do not use the block passed to them, and to error if a block was passed to such methods. As far as I can tell, it was later on closed since #10499 solved a large part of the problem. That proposal has, as part of the dev meeting discussion, a proposal from @matz to allow methods to use `&nil` to explicitly declare that they don't accept a block. At the time, the proposal was trying to solve a bigger problem, so this sub-proposal was never considered seriously. However, notes in the proposal say: > It is explicit, but it is tough to add this `&nil` parameter declaration to all of methods (do you want to add it to `def []=(i, e, &nil)`?). (I agree `&nil` is valuable on some situations) This proposal extracts that sub-proposal to make this a new language feature. ## Proposal In Ruby, it is always valid for the caller to pass a block to a method call, even if the callee is not expecting a block to be passed. This leads to subtle user errors, where the author of some code assumes a method call uses a block, but the block passed to the method call is silently ignored. The proposal is to introduce `&nil` at method declaration sites to mean "This method does not accept a block". This is symmetric to the ability to pass `&nil` at call sites to mean "I am not passing a block to this method call", which is sometimes useful when making `super` calls (since blocks are always implicitly passed). Explicitly, the proposal is to make the following behaviour be a part of Ruby: ```ruby def find(item = nil, &nil) # some implementation that doesn't call `yield` or `block_given?` end find { |i| i == 42 } # => ArgumentError: passing block to the method `find' that does not accept a block. ``` ## Implementation I assume the implementation would be a grammar change to make `&nil` valid at method declaration sites, as well as raising an `ArgumentError` for methods that are called with a block but are declared with `&nil`. ## Evaluation Since I don't have an implementation, I can't make a proper evaluation of the feature proposal. However, I would expect the language changes to be minimal with no runtime costs for methods that don't use the `&nil` syntax. ## Discussion This proposal has much smaller scope than #15554 so that the Ruby language can start giving library authors the ability to explicitly mark their methods as not accepting a block. This is fully backward compatible, since it is an opt-in behaviour and not an opt-out one. Future directions after this feature proposal could be a way to signal to the VM that any method in a file that doesn't explicitly use `yield`/`block_given?` or explicitly declared a block parameter should be treated as not accepting a block. This can be done via some kind of pragma similar to `frozen_string_literal`, or through other means. However, such future directions are beyond the scope of this proposal. ## Summary Adding the ability for methods to declare that they don't accept a block will make writing code against such methods safer and more resilient, and will prevent silently ignored behaviour that is often hard to catch or troubleshoot. -- https://bugs.ruby-lang.org/

9 20

[ruby-core:111448] [Ruby master Bug#19266] URI::Generic should use URI::RFC3986_PARSER instead of URI::DEFAULT_PARSER
by gareth (Gareth Adams) 22 Jul '24

22 Jul '24

Issue #19266 has been reported by gareth (Gareth Adams). ---------------------------------------- Bug #19266: URI::Generic should use URI::RFC3986_PARSER instead of URI::DEFAULT_PARSER https://bugs.ruby-lang.org/issues/19266 * Author: gareth (Gareth Adams) * Status: Open * Priority: Normal * ruby -v: ruby 3.1.3p185 (2022-11-24 revision 1a6b16756e) [arm64-darwin21] * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- In June 2014, [`uri/common` was updated][1] to introduce a RFC3986-compliant parser (`URI::RFC3986_PARSER`) as an alternative to the previous RFC2396 parser, and common methods like `URI()` were updated to use that new parser by default. The only methods in `common` not updated were [`URI.extract` and `URI.regexp`][2] which are marked as obsolete. (The old parser was kept in the `DEFAULT_PARSER` constant despite it not being the default for those methods, presumably for backward compatibility.) However, similar [methods called on `URI::Generic`][3] were never updated to use this new parser. This means that methods like `URI::Generic.build` fail when given input that succeeds normally, and this also affects subclasses like URI::HTTP: ``` $ pry -r uri -r uri/common -r uri/generic [1] pry(main)> URI::Generic.build(host: "underscore_host.example") URI::InvalidComponentError: bad component(expected host component): underscore_host.example from /Users/gareth/.asdf/installs/ruby/3.1.3/lib/ruby/3.1.0/uri/generic.rb:591:in `check_host' [2] pry(main)> URI::HTTP.build(host: "underscore_host.example") URI::InvalidComponentError: bad component(expected host component): underscore_host.example from /Users/gareth/.asdf/installs/ruby/3.1.3/lib/ruby/3.1.0/uri/generic.rb:591:in `check_host' [3] pry(main)> URI("http://underscore_host.example") => #<URI::HTTP http://underscore_host.example> ``` `URI::Generic.new` allows a configurable `parser` positional argument to override the class' default parser, but other factory methods like `.build` don't allow this override. Arguably this doesn't cause problems because at least in the case above, the URI can be built with the polymorphic constructor, but having the option to build URIs from explicit named parts is useful, and leaving the outdated functionality in the `Generic` class is ambiguous. It's possible that the whole Generic class and its subclasses aren't intended to be used directly how I'm intending here, but there's nothing I could see that suggested this is the case. I'm not aware of the entire list of differences between RFC2396 and RFC3986. The relevant difference here is that in RFC2396 an individual segment of a host ([`domainlabel`s][4]) could only be `alphanum | alphanum *( alphanum | "-" ) alphanum`, whereas RFC3986 allows [hostnames][5] to include any of `ALPHA / DIGIT / "-" / "." / "_" / "~"`. It's possible that other differences might cause issues for developers, but since this has gone over 8 years without anyone else caring about this, this is definitely not especially urgent. [1]: https://github.com/ruby/ruby/commit/bb83f32dc3e0424d25fa4e55d8ff32b061320e41 [2]: https://github.com/ruby/ruby/blob/28a17436503c3c4cb7a35b423a894b697cd80da9/… [3]: https://github.com/ruby/ruby/blob/28a17436503c3c4cb7a35b423a894b697cd80da9/… [4]: https://www.rfc-editor.org/rfc/rfc2396#section-3.2.2 [5]: https://www.rfc-editor.org/rfc/rfc3986#page-13 -- https://bugs.ruby-lang.org/

4 8

[ruby-core:113352] [Ruby master Bug#19619] Endless method definition with parameters for an instance given as numbered parameters doesn't work
by okuramasafumi (Masafumi OKURA) 21 Jul '24

21 Jul '24

Issue #19619 has been reported by okuramasafumi (Masafumi OKURA). ---------------------------------------- Bug #19619: Endless method definition with parameters for an instance given as numbered parameters doesn't work https://bugs.ruby-lang.org/issues/19619 * Author: okuramasafumi (Masafumi OKURA) * Status: Open * Priority: Normal * ruby -v: 3.2.2 * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- This works: ```ruby class Foo def bar(baz) = 'foo' end p Foo.new.bar('baz') ``` This also works: ```ruby o = Object.new o.tap { |obj| def obj.bar(baz) = 'foo' } p o.bar('baz') ``` Even this works: ```ruby o = Object.new o.tap { def _1.bar = 'foo' } p o.bar ``` But this DOESN'T work: ```ruby o = Object.new o.tap { def _1.bar(baz) = 'foo' } p o.bar('baz') ``` So, when we define a method with parameters on an instance given as numbered parameters, it doesn't work. If we change one condition (definition way, parameter existence, and so on), it works. -- https://bugs.ruby-lang.org/

4 4

[ruby-core:114792] [Ruby master Bug#19887] RUBYOPT doesn't work unless starting with `-`
by nobu (Nobuyoshi Nakada) 20 Jul '24

20 Jul '24

Issue #19887 has been reported by nobu (Nobuyoshi Nakada). ---------------------------------------- Bug #19887: RUBYOPT doesn't work unless starting with `-` https://bugs.ruby-lang.org/issues/19887 * Author: nobu (Nobuyoshi Nakada) * Status: Assigned * Priority: Normal * Assignee: nobu (Nobuyoshi Nakada) * Backport: 3.0: REQUIRED, 3.1: REQUIRED, 3.2: REQUIRED ---------------------------------------- It was working till 1.8. ```sh-session $ for ruby in ruby{1.{8..9},2.{0..7},3.{0..3}}; do RUBYOPT=w $ruby -e 'puts "'$ruby': #{$VERBOSE}"'; done ruby1.8: true ruby1.9: false ruby2.0: false ruby2.1: false ruby2.2: false ruby2.3: false ruby2.4: false ruby2.5: false ruby2.6: false ruby2.7: false ruby3.0: false ruby3.1: false ruby3.2: false ruby3.3: false ``` -- https://bugs.ruby-lang.org/

2 2

[ruby-core:115728] [Ruby master Bug#20062] Numbered parameters are broken in Ruby 3.3-dev
by osyo (manga osyo) 20 Jul '24

20 Jul '24

Issue #20062 has been reported by osyo (manga osyo). ---------------------------------------- Bug #20062: Numbered parameters are broken in Ruby 3.3-dev https://bugs.ruby-lang.org/issues/20062 * Author: osyo (manga osyo) * Status: Open * Priority: Normal * ruby -v: ruby 3.3.0dev (2023-12-13T13:16:11Z master f390c51b15) [x86_64-linux] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Define Numbered parameters and a method in one block and then reference Numbered parameters in another block, you will get a SyntaxError in Ruby 3.3. This is not a problem in Ruby 3.2. ```ruby 'a'.tap do p _1 def f1; end end 'a'.tap do p _1 end __END__ output: Ruby 3.2.2 => "a" "a" Ruby 3.3.0dev(2023-12-13T13:16:11Z master f390c51b15) => /path/to/test.rb: /path/to/test.rb:8: numbered parameter is already used in (SyntaxError) /path/to/test.rb:2: outer block here ``` I noticed this while checking the operation of https://bugs.ruby-lang.org/issues/18966. I haven't checked, but https://bugs.ruby-lang.org/issues/19619 may be related. -- https://bugs.ruby-lang.org/

4 3

[ruby-core:114258] [Ruby master Bug#19781] Stack consistency error with tailcall and YJIT enabled
by k0kubun (Takashi Kokubun) 20 Jul '24

20 Jul '24

Issue #19781 has been reported by k0kubun (Takashi Kokubun). ---------------------------------------- Bug #19781: Stack consistency error with tailcall and YJIT enabled https://bugs.ruby-lang.org/issues/19781 * Author: k0kubun (Takashi Kokubun) * Status: Open * Priority: Normal * Assignee: yjit * Backport: 3.0: DONTNEED, 3.1: REQUIRED, 3.2: REQUIRED ---------------------------------------- ```rb src = <<-EOS def apply_one_and_two(&block) [1, p(1)] end def add_one_and_two apply_one_and_two(&:+) end EOS RubyVM::InstructionSequence.new( "proc {|_|_.class_eval {#{src}}}", __FILE__, __FILE__, 1, tailcall_optimization: true, trace_instruction: false, ).eval[self.singleton_class] def entry add_one_and_two end entry ``` This script causes "Stack consistency error" with `ruby --yjit-call-threshold=1` on any YJIT-enabled build (3.1, 3.2, and master). -- https://bugs.ruby-lang.org/

2 1

[ruby-core:115549] [Ruby master Bug#20030] `Ripper.tokenize('"\\C-あ"')` separates encoding valid string to encoding invalid string.
by tompng (tomoya ishida) 15 Jul '24

15 Jul '24

Issue #20030 has been reported by tompng (tomoya ishida). ---------------------------------------- Bug #20030: `Ripper.tokenize('"\\C-あ"')` separates encoding valid string to encoding invalid string. https://bugs.ruby-lang.org/issues/20030 * Author: tompng (tomoya ishida) * Status: Open * Priority: Normal * ruby -v: ruby 3.3.0dev (2023-11-30T16:23:25Z master d048bae96b) [x86_64-linux] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- ~~~ruby Ripper.tokenize '"\\C-あ"' # or Ripper.tokenize "\"\\C-\u3042\"" # => ["\"", "\x81", "\x82", "\""] ~~~ I expect all tokens to be valid_encoding if the source string is valid_encoding. This is causing IRB crash when typing `"\C-あ"`. -- https://bugs.ruby-lang.org/

3 2

[ruby-core:116114] [Ruby master Bug#20169] `GC.compact` can raises `EFAULT` on IO
by ko1 (Koichi Sasada) 15 Jul '24

15 Jul '24

Issue #20169 has been reported by ko1 (Koichi Sasada). ---------------------------------------- Bug #20169: `GC.compact` can raises `EFAULT` on IO https://bugs.ruby-lang.org/issues/20169 * Author: ko1 (Koichi Sasada) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- 1. `GC.compact` introduces read barriers to detect read accesses to the pages. 2. I/O operations release GVL to pass the control while their execution, and another thread can call `GC.compact` (or auto compact feature I guess, but not checked yet). 3. Call `write(ptr)` can return `EFAULT` when `GC.compact` is running because `ptr` points read-barrier protected pages. Reproducible steps: Apply the following patch to increase possibility: ```patch diff --git a/io.c b/io.c index f6cd2c1a56..83d67ba2dc 100644 --- a/io.c +++ b/io.c @@ -1212,8 +1212,12 @@ internal_write_func(void *ptr) } } + int cnt = 0; retry: - do_write_retry(write(iis->fd, iis->buf, iis->capa)); + for (; cnt < 1000; cnt++) { + do_write_retry(write(iis->fd, iis->buf, iis->capa)); + if (result <= 0) break; + } if (result < 0 && !iis->nonblock) { int e = errno; ``` Run the following code: ```ruby t1 = Thread.new{ 10_000.times.map{"#{_1}"}; GC.compact while true } t2 = Thread.new{ i=0 $stdout.write "<#{i+=1}>" while true } t2.join ``` and ``` $ make run (snip) 4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4><4>#<Thread:0x00007fa61b4dd758 ../../src/trunk/test.rb:3 run> terminated with exception (report_on_exception is true): ../../src/trunk/test.rb:5:in `write': Bad address @ io_write - <STDOUT> (Errno::EFAULT) from ../../src/trunk/test.rb:5:in `block in <main>' ../../src/trunk/test.rb:5:in `write': Bad address @ io_write - <STDOUT> (Errno::EFAULT) from ../../src/trunk/test.rb:5:in `block in <main>' make: *** [uncommon.mk:1383: run] Error 1 ``` I think this is why we get `EFAULT` on CI. To increase possibilities running many busy processes (`ruby -e 'loop{}'` for example) will help (and on CI environment there are such busy processes accidentally). -- https://bugs.ruby-lang.org/

6 13

[ruby-core:116078] [Ruby master Bug#20162] Memory leak when duplicating too complex object
by peterzhu2118 (Peter Zhu) 08 Jul '24

08 Jul '24

Issue #20162 has been reported by peterzhu2118 (Peter Zhu). ---------------------------------------- Bug #20162: Memory leak when duplicating too complex object https://bugs.ruby-lang.org/issues/20162 * Author: peterzhu2118 (Peter Zhu) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: DONTNEED, 3.2: REQUIRED, 3.3: REQUIRED ---------------------------------------- GitHub PR: https://github.com/ruby/ruby/pull/9449 Creating a ST table then calling st_replace leaks memory because the st_replace overwrites the ST table without freeing any of the existing memory. This commit changes it to use st_copy instead. For example: ```ruby RubyVM::Shape.exhaust_shapes o = Object.new o.instance_variable_set(:@a, 0) 10.times do 100_000.times { o.dup } puts `ps -o rss= -p #{$$}` end ``` Before: ``` 23264 33600 42672 52160 61600 71728 81056 90528 100560 109840 ``` After: ``` 14752 14816 15584 15584 15664 15664 15664 15664 15664 15664 ``` -- https://bugs.ruby-lang.org/

3 2

[ruby-core:114932] [Ruby master Bug#19907] Method calls with keyword arguments in eval leaks callcache and callinfo objects
by peterzhu2118 (Peter Zhu) 07 Jul '24

07 Jul '24

Issue #19907 has been reported by peterzhu2118 (Peter Zhu). ---------------------------------------- Bug #19907: Method calls with keyword arguments in eval leaks callcache and callinfo objects https://bugs.ruby-lang.org/issues/19907 * Author: peterzhu2118 (Peter Zhu) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- The following script leaks callcache and callinfo objects: ```ruby def foo(a:); end 10.times do 10_000.times do eval(<<~RUBY) foo(a: 1) RUBY end puts "Number of live objects: #{GC.stat(:heap_live_slots)}" puts "Memory usage: #{`ps -o rss= -p #{$$}`}" puts end ``` Output is: ``` Number of live objects: 46248 Memory usage: 16160 Number of live objects: 65902 Memory usage: 19888 Number of live objects: 92656 Memory usage: 24032 Number of live objects: 126791 Memory usage: 28048 Number of live objects: 132919 Memory usage: 28816 Number of live objects: 180687 Memory usage: 32384 Number of live objects: 181957 Memory usage: 32464 Number of live objects: 227485 Memory usage: 34224 Number of live objects: 256101 Memory usage: 37200 Number of live objects: 274151 Memory usage: 38752 ``` After performing a `ObjectSpace.dump_all`, I found that it is leaking callcache and callinfo objects that is being held on by the `Object` class. ```json {"address":"0x102ecef70", "type":"CLASS", "shape_id":2, "slot_size":160, "class":"0x102ecf8d0", "variation_count":0, "superclass":"0x102ecfd30", "real_class_name":"Object", "singleton":true, "references":[ ... ]} {"address":"0x1030c90a0", "type":"IMEMO", "shape_id":0, "slot_size":40, "imemo_type":"callinfo", "mid":"foo", "memsize":40, "flags":{"wb_protected":true}} {"address":"0x102fad568", "type":"IMEMO", "shape_id":0, "slot_size":40, "imemo_type":"callcache", "memsize":40, "flags":{"wb_protected":true, "old":true, "uncollectible":true, "marked":true}} ``` -- https://bugs.ruby-lang.org/

5 4