March 2024 - ruby-core - ml.ruby-lang.org

[ruby-core:116073] [Ruby master Bug#20161] Memory leak in regexp grapheme clusters
by peterzhu2118 (Peter Zhu) 14 Mar '24

14 Mar '24

Issue #20161 has been reported by peterzhu2118 (Peter Zhu). ---------------------------------------- Bug #20161: Memory leak in regexp grapheme clusters https://bugs.ruby-lang.org/issues/20161 * Author: peterzhu2118 (Peter Zhu) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: REQUIRED, 3.2: REQUIRED, 3.3: REQUIRED ---------------------------------------- GitHub PR: https://github.com/ruby/ruby/pull/9447 The cc->mbuf gets overwritten, so we need to free it to not leak memory. For example: ```ruby str = "hello world".encode(Encoding::UTF_32LE) 10.times do 1_000.times do str.grapheme_clusters end puts `ps -o rss= -p #{$$}` end ``` Before: ``` 15536 15760 15920 16144 16304 16480 16640 16784 17008 17280 ``` After: ``` 15584 15584 15760 15824 15888 15888 15888 15888 16048 16112 ``` -- https://bugs.ruby-lang.org/

3 2

[ruby-core:117129] [Ruby master Bug#20333] segfault while running my tests
by dorianmariefr 14 Mar '24

14 Mar '24

Issue #20333 has been reported by dorianmariefr (Dorian Marié). ---------------------------------------- Bug #20333: segfault while running my tests https://bugs.ruby-lang.org/issues/20333 * Author: dorianmariefr (Dorian Marié) * Status: Open * ruby -v: ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [arm64-darwin23] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- No sure how to debug it, I had a segfault while running the tests, maybe you will find it useful. ---Files-------------------------------- segfault.log (731 KB) crash.log (100 KB) -- https://bugs.ruby-lang.org/

2 3

[ruby-core:117155] [Ruby master Feature#4247] New features for Array#sample, Array#choice
by mame (Yusuke Endoh) 14 Mar '24

14 Mar '24

Issue #4247 has been updated by mame (Yusuke Endoh). Status changed from Assigned to Rejected Assignee deleted (mame (Yusuke Endoh)) We discussed this at the dev meeting. No one remembered the discussion from over 10 years ago, so we discussed it anew and concluded that this was a no-go. A naive API design could be `ary.sample(k, weights: [Float])`, but this would be an O(ary.size * k) time-consuming algorithm. There are many more efficient algorithms for weighted sampling. (We read Julia's [StatsBase.jl](https://juliastats.org/StatsBase.jl/stable/sampling/) and Python's [random](https://docs.python.org/3.13/library/random.html).) However, these require additional information, such as the sum of the weights, cumulative weight table, the need to build the table in advance, etc. We want to avoid an API design that only allows slow algorithm, but it seems overkill to introduce an API that allows advanced algorithms as a built-in feature. We concluded that it would be better to make a gem, instead of a built-in feature, for such things. ---------------------------------------- Feature #4247: New features for Array#sample, Array#choice https://bugs.ruby-lang.org/issues/4247#change-107242 * Author: oj (Yoji Ojima) * Status: Rejected ---------------------------------------- =begin We are planning to add the following features of the random sampling to Array. 1. Weighted random sampling. 2. Sampling with replacement. 3. Iteration. It is discussed in ruby-dev (Feature #3647 and #4147). API will be: Array#sample([size, [opt]]) - Random selection without replacement. - Returns a new array when size is specified. - opt: weight: proc or array random: Random instance Array#choice([size, [opt]]) - Random selection with replacement. - Returns a new array when size is specified. - opt: same as above. Array#each_sample([opt]) - Random selection iterator without replacement. - Choose a random element and yield it. - Returns an Enumerator if a block is not given. - opt: same as above. Array#each_choice([opt]) - Random selection iterator with replacement. - Choose a random element and yield it. - Returns an Enumerator if a block is not given. - opt: same as above. Comments? =end -- https://bugs.ruby-lang.org/

1 0

[ruby-core:116684] [Ruby master Feature#20261] Add symbol alternatives to '' and nil for IO line separators
by burdettelamar (Burdette Lamar) 14 Mar '24

14 Mar '24

Issue #20261 has been reported by burdettelamar (Burdette Lamar). ---------------------------------------- Feature #20261: Add symbol alternatives to '' and nil for IO line separators https://bugs.ruby-lang.org/issues/20261 * Author: burdettelamar (Burdette Lamar) * Status: Open * Priority: Normal ---------------------------------------- For IO's line-oriented read methods, there are two special values for the line-separator argument `sep`; I'm proposing to add (user-friendlier) symbol synonyms for those values: - `:paragraph` as synonym for `''` (read paragraphs). - `:slurp` as synonym for `nil` (read all). Details (code, documentation, tests) may be seen at https://github.com/ruby/ruby/pull/9921. -- https://bugs.ruby-lang.org/

2 2

[ruby-core:116987] [Ruby master Bug#20310] ASAN fake stacks need to be marked during GC for non-current execution context
by kjtsanaktsidis (KJ Tsanaktsidis) 14 Mar '24

14 Mar '24

Issue #20310 has been reported by kjtsanaktsidis (KJ Tsanaktsidis). ---------------------------------------- Bug #20310: ASAN fake stacks need to be marked during GC for non-current execution context https://bugs.ruby-lang.org/issues/20310 * Author: kjtsanaktsidis (KJ Tsanaktsidis) * Status: Assigned * Assignee: kjtsanaktsidis (KJ Tsanaktsidis) * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- When ASAN (address sanitizer) is enabled, the compiler does not necessarily store all local variables on the real machine stack; instead, locals can be stored in per-frame heap allocated memory which ASAN uses to detect things like stack-use-after-return ("fake stacks"). A pointer to the fake stack is left on the real machine stack, so it's possible to discover these fake stacks during GC and mark locals stored there as well. At the moment, Ruby is currently marking these fake stacks for the current execution context which triggered GC, as part of `mark_current_machine_context`: https://github.com/ruby/ruby/blob/fe0b704df5553bdd59e90650ffbbfac785a2e48a/…. However, there are other machine stacks which also need to be marked like this: * Machine stacks for other threads which did not trigger GC are marked in `rb_execution_context_mark` here: https://github.com/ruby/ruby/blob/fe0b704df5553bdd59e90650ffbbfac785a2e48a/… * Machine stacks for fibers are marked in `cont_mark` here: https://github.com/ruby/ruby/blob/fe0b704df5553bdd59e90650ffbbfac785a2e48a/… We need to make these two kinds of stacks perform the same ASAN fake stack marking as `mark_current_machine_context` does. (P.S. - `callcc` continuations are another kind of machine stack which get marked, but ASAN is not compatible with callcc, so this doesn't really matter). (P.S. - it appears to me that the currently-switched-to fiber will have its stack marked _twice_; once in `rb_execution_context_mark` or `mark_current_machine_context, and once in `cont_mark`; if this is true, I will fix this too) -- https://bugs.ruby-lang.org/

2 2

[ruby-core:116963] [Ruby master Feature#20306] Add rb_free_at_exit_p
by peterzhu2118 (Peter Zhu) 14 Mar '24

14 Mar '24

Issue #20306 has been reported by peterzhu2118 (Peter Zhu). ---------------------------------------- Feature #20306: Add rb_free_at_exit_p https://bugs.ruby-lang.org/issues/20306 * Author: peterzhu2118 (Peter Zhu) * Status: Open ---------------------------------------- GitHub PR: https://github.com/ruby/ruby/pull/10104 From ticket [#20290](https://bugs.ruby-lang.org/issues/20290#note-6), I found that C extensions could use ruby_vm_at_exit to register hooks to free memory at shutdown. However, they cannot determine whether they should free all memory during shutdown to mirror the behavior of Ruby when RUBY_FREE_AT_EXIT is set. This ticket proposes a new API called rb_free_at_exit_p that returns true when RUBY_FREE_AT_EXIT is set, and false otherwise. -- https://bugs.ruby-lang.org/

3 4

[ruby-core:116909] [Ruby master Feature#20293] Add `Warning.categories` method that returns the warning category names
by nobu (Nobuyoshi Nakada) 14 Mar '24

14 Mar '24

Issue #20293 has been reported by nobu (Nobuyoshi Nakada). ---------------------------------------- Feature #20293: Add `Warning.categories` method that returns the warning category names https://bugs.ruby-lang.org/issues/20293 * Author: nobu (Nobuyoshi Nakada) * Status: Open ---------------------------------------- I propose a new method `Warning.categories`. This would be useful (or necessary) for tests mainly. Currently, `EnvUtil.capture_global_values` saves the original warning settings as followings: ```ruby @original_warning = defined?(Warning.[]) ? %i[deprecated experimental].to_h {|i| [i, Warning[i]]} : nil ``` But this is wrong now; `performance` is missing. So we need: ```ruby @original_warning = if defined?(Warning.[]) # 2.7+ %i[deprecated experimental performance].to_h do |i| [i, begin Warning[i]; rescue ArgumentError; end] end.compact end ``` That means this list is version dependent and we will need to maintain this list in future. We need another surefire way. -- https://bugs.ruby-lang.org/

1 1

[ruby-core:116489] [Ruby master Bug#20224] Backport https://github.com/ruby/ruby/pull/9634 to Ruby 3.3
by yui-knk (Kaneko Yuichiro) 14 Mar '24

14 Mar '24

Issue #20224 has been reported by yui-knk (Kaneko Yuichiro). ---------------------------------------- Bug #20224: Backport https://github.com/ruby/ruby/pull/9634 to Ruby 3.3 https://bugs.ruby-lang.org/issues/20224 * Author: yui-knk (Kaneko Yuichiro) * Status: Closed * Priority: Normal * Backport: 3.0: DONTNEED, 3.1: DONTNEED, 3.2: DONTNEED, 3.3: REQUIRED ---------------------------------------- It's nice if we can backport https://github.com/ruby/ruby/pull/9634 (3d19409637de1462b6790d2a92344bf0a10d8c52) to Ruby 3.3 branch. Even so previous codes work as expected but it depends on how parse.c works and misleading. -- https://bugs.ruby-lang.org/

2 1

[ruby-core:117139] [Ruby master Feature#15554] warn/error passing a block to a method which never use a block
by ko1 (Koichi Sasada) 14 Mar '24

14 Mar '24

Issue #15554 has been updated by ko1 (Koichi Sasada). Matz accepted to try it with `-w` (WOW!) so I'll make a patch. ---------------------------------------- Feature #15554: warn/error passing a block to a method which never use a block https://bugs.ruby-lang.org/issues/15554#change-107218 * Author: ko1 (Koichi Sasada) * Status: Open * Assignee: matz (Yukihiro Matsumoto) ---------------------------------------- # Abstract Warn or raise an ArgumentError if block is passed to a method which does not use a block. In other words, detect "block user methods" implicitly and only "block user methods" can accept a block. # Background Sometimes, we pass a block to a method which ignores the passed block accidentally. ``` def my_open(name) open(name) end # user hopes it works as Kernel#open which invokes a block with opened file. my_open(name){|f| important_work_with f } # but simply ignored... ``` To solve this issue, this feature request propose showing warnings or raising an exception on such case. Last developer's meeting, matz proposed `&nil` which declares this method never receive a block. It is explicit, but it is tough to add this `&nil` parameter declaration to all of methods (do you want to add it to `def []=(i, e, &nil)`?). (I agree `&nil` is valuable on some situations) # Spec ## Define "use a block" methods We need to define which method accepts a block and which method does not. * (1) method has a block parameter (`&b`) * (2) method body has `yield' * (3) method body has `super` (ZSUPER in internal terminology) or `super(...)` * (4) method body has singleton method (optional) (1) and (2) is very clear. I need to explain about (3) and (4). (3). `super` (ZSUPER) passes all parameters as arguments. So there is no surprise that which can accept `block`. However `super(...)` also passes a block if no explicit block passing (like `super(){}` or `super(&b)`) are written. I'm not sure we need to continue this strange specification, but to keep compatibility depending this spec, I add this rule. (4). surprisingly, the following code invoke a block: ``` def foo class << Object.new yield end end foo{ p :ok } #=> :ok ``` I'm also not sure we need to keep this spec, but to allow this spec, I added (4) rule. Strictly speaking, it is not required, but we don't keep the link from singleton class ISeq to lexical parent iseq now, so I added it. ## Exceptional cases A method called by `super` doesn`t warn warning even if this method doesn't use a block. The rule (3) can pass blocks easily and there are many methods don`t use a block. So my patch ignores callings by `super`. ## corner cases There are several cases to use block without (1)-(4) rules. ### `Proc.new/proc/lambda` without a block Now it was deprecated in r66772 (commit:9f1fb0a17febc59356d58cef5e98db61a3c03550). Related discussion: [Bug #15539] ### `block_given?` `block_given?` expects block, but I believe we use it with `yield` or a block parameter. If you know the usecase without them, please tell us. ### `yield` in `eval` We can't know `yield` (or (3), (4) rule) in an `eval` evaluating string at calling time. ``` def foo eval('yield`) end foo{} # at calling time, # we can't know the method foo can accept a block or not. ``` So I added a warning to use `yield` in `eval` like that: `test.rb:4: warning: use yield in eval will not be supported in Ruby 3.` Workaround is use a block parameter explicitly. ``` def foo &b eval('b.call') end foo{ p :ok } ``` # Implementation Strategy is: * [compile time] introduce `iseq::has_yield` field and check it if the iseq (or child iseq) contains `yield` (or something) * [calling time] if block is given, check `iseq::has_yield` flag and show warning (or raise an exception) https://gist.github.com/ko1/c9148ad0224bf5befa3cc76ed2220c0b On this patch, now it raises an error to make it easy to detect. It is easy to switch to show the warning. # Evaluation and discussion I tried to avoid ruby's tests. https://gist.github.com/ko1/37483e7940cdc4390bf8eb0001883786 Here is a patch. There are several patterns to avoid warnings. ## tests for `block_given?`, `Proc.new` (and similar) without block Add a dummy block parameter. It is test-specific issue. ## empty `each` Some tests add `each` methods do not `yield`, like: `def each; end`. Maybe test-specific issue, and adding a dummy block parameter. ## Subtyping / duck typing https://github.com/ruby/ruby/blob/c01a5ee85e2d6a7128cccafb143bfa694284ca87/… This `parse` method doesn't use `yield`, but other sub-type's `parse` methods use. ## `super` with `new` method https://gist.github.com/ko1/37483e7940cdc4390bf8eb0001883786#file-tests-pat… This method override `Class#new` method and introduce a hook with block (yield a block in this hook code). https://github.com/ruby/ruby/blob/trunk/lib/rubygems/package/tar_writer.rb#… In this method, call `super` and it also passing a block. However, called `initialize` doesn't use a block. ## Change robustness This change reduce robustness for API change. `Delegator` requires to support `__getobj__` for client classes. Now `__getobj__` should accept block but most of `__getobj__` clients do not call given block. https://github.com/ruby/ruby/blob/trunk/lib/delegate.rb#L80 This is because of delegator.rb's API change. https://gist.github.com/ko1/37483e7940cdc4390bf8eb0001883786#file-tests-pat… Nobu says calling block is not required (ignoring a block is no problem) so it is not a bug for delegator client classes. ## Found issues. ``` [ 2945/20449] Rinda::TestRingServer#test_do_reply = 0.00 s 1) Error: Rinda::TestRingServer#test_do_reply: ArgumentError: passing block to the method "with_timeout" (defined at /home/ko1/src/ruby/trunk/test/rinda/test_rinda.rb:787) is never used. /home/ko1/src/ruby/trunk/test/rinda/test_rinda.rb:635:in `test_do_reply' [ 2946/20449] Rinda::TestRingServer#test_do_reply_local = 0.00 s 2) Error: Rinda::TestRingServer#test_do_reply_local: ArgumentError: passing block to the method "with_timeout" (defined at /home/ko1/src/ruby/trunk/test/rinda/test_rinda.rb:787) is never used. /home/ko1/src/ruby/trunk/test/rinda/test_rinda.rb:657:in `test_do_reply_local' [10024/20449] TestGemRequestSetGemDependencyAPI#test_platform_mswin = 0.01 s 3) Error: TestGemRequestSetGemDependencyAPI#test_platform_mswin: ArgumentError: passing block to the method "util_set_arch" (defined at /home/ko1/src/ruby/trunk/lib/rubygems/test_case.rb:1053) is never used. /home/ko1/src/ruby/trunk/test/rubygems/test_gem_request_set_gem_dependency_api.rb:655:in `test_platform_mswin' [10025/20449] TestGemRequestSetGemDependencyAPI#test_platforms = 0.01 s 4) Error: TestGemRequestSetGemDependencyAPI#test_platforms: ArgumentError: passing block to the method "util_set_arch" (defined at /home/ko1/src/ruby/trunk/lib/rubygems/test_case.rb:1053) is never used. /home/ko1/src/ruby/trunk/test/rubygems/test_gem_request_set_gem_dependency_api.rb:711:in `test_platforms' ``` These 4 detection show the problem. `with_timeout` method (used in Rinda test) and `util_set_arch` method (used in Rubygems test) simply ignore the given block. So these tests are simply ignored. I reported them. (https://github.com/rubygems/rubygems/issues/2601) ## raise an error or show a warning? At least, Ruby 2.7 should show warning for this kind of violation with `-w`. How about for Ruby3? -- https://bugs.ruby-lang.org/

1 0

[ruby-core:116846] [Ruby master Misc#20281] DevMeeting-2024-03-14
by mame (Yusuke Endoh) 12 Mar '24

12 Mar '24

Issue #20281 has been reported by mame (Yusuke Endoh). ---------------------------------------- Misc #20281: DevMeeting-2024-03-14 https://bugs.ruby-lang.org/issues/20281 * Author: mame (Yusuke Endoh) * Status: Open * Priority: Normal ---------------------------------------- # The next dev meeting **Date: 2024/03/14 13:00-17:00** (JST) Log: *TBD* - Dev meeting *IS NOT* a decision-making place. All decisions should be done at the bug tracker. - Dev meeting is a place we can ask Matz, nobu, nurse and other developers directly. - Matz is a very busy person. Take this opportunity to ask him. If you can not attend, other attendees can ask instead of you (if attendees can understand your issue). - We will write a record of the discussion in the file or to each ticket in English. - All activities are best-effort (keep in mind that most of us are volunteer developers). - The date, time and place of the meeting are scheduled according to when/where we can reserve Matz's time. - *DO NOT* discuss then on this ticket, please. # Call for agenda items If you have a ticket that you want matz and committers to discuss, please post it into this ticket in the following format: ``` * [Ticket ref] Ticket title (your name) * Comment (A summary of the ticket, why you put this ticket here, what point should be discussed, etc.) ``` Example: ``` * [Feature #14609] `Kernel#p` without args shows the receiver (ko1) * I feel this feature is very useful and some people say :+1: so let discuss this feature. ``` - It is recommended to add a comment by 2024/03/11. We hold a preparatory meeting to create an agenda a few days before the dev-meeting. - The format is strict. We'll use [this script to automatically create an markdown-style agenda](https://gist.github.com/mame/b0390509ce1491b43610b9ebb665eb86). We may ignore a comment that does not follow the format. - Your comment is mandatory. We cannot read all discussion of the ticket in a limited time. We appreciate it if you could write a short summary and update from a previous discussion. -- https://bugs.ruby-lang.org/

14 15