November 2024 - ruby-core - ml.ruby-lang.org

[ruby-core:118243] [Ruby master Feature#20564] Switch default parser to Prism
by kddnewton (Kevin Newton) 24 Dec '24

24 Dec '24

Issue #20564 has been reported by kddnewton (Kevin Newton). ---------------------------------------- Feature #20564: Switch default parser to Prism https://bugs.ruby-lang.org/issues/20564 * Author: kddnewton (Kevin Newton) * Status: Open ---------------------------------------- This issue is to propose switching the default parser for Ruby to Prism on the master branch. The main goal is to get this default into both master and the next preview so that the community has plenty of time to test before the eventual 3.4 release. Matz has indicated that the final decision about the default parser for Ruby 3.4 will be made later in the year. Decisions about the default parser for Ruby 3.5+ and the relationship between Prism, lrama, parse.y, etc. will also be made at a later time. ## Background For some background on the Prism project itself, here are a couple of links in chronological order that you can read: * https://bugs.ruby-lang.org/issues/19741 * https://railsatscale.com/2023-06-12-rewriting-the-ruby-parser/ * https://kddnewton.com/2024/01/23/prism.html * https://railsatscale.com/2024-04-16-prism-in-2024/ ## Status Prism now passes all Ruby tests and specs. Internally at Shopify, it also passes our Core monolith's CI, which is a large and complex Rails application. We are confident that it is ready for production use. External to CRuby, Prism has already been adopted extensively around the ecosystem, including but not limited to: * [JRuby](https://github.com/jruby/jruby/pull/8103) (via Java) * [TruffleRuby](https://github.com/oracle/truffleruby/issues/3117) (via Java) * [Natalie](https://github.com/natalie-lang/natalie/pull/1213) (via C++ and Ruby) * [Opal](https://github.com/opal/opal/pull/2642) (via Ruby and WASM) * [repl_type_completor](https://github.com/ruby/repl_type_completor) * [rubocop](https://docs.rubocop.org/rubocop/configuration.html#setting-the-pa… (via parser translator) * [ruby-lsp](https://github.com/Shopify/ruby-lsp) * [packwerk](https://github.com/Shopify/packwerk/pull/388) (via parser translator) Fortunately with the community adoption, we have been able to find many edge cases and bugs that have been fixed. Thank you so much to everyone who has tried, used, reported problems with, and improved Prism in the last couple of years. We would not have hit this milestone without this community involvement. ## Caveats While Prism correctly accepts and parses all valid Ruby syntax, there are still a number of cases where it does not reject invalid syntax as it should. We are currently working on these cases, and we are confident that we can eliminate them before the next preview release. You can find them listed here: https://github.com/ruby/prism/issues?q=is%3Aopen+is%3Aissue+label%3Ainvalid…. This is our top priority right now, and I anticipate these being resolved quickly. The other caveat is unknown unknowns: bugs or syntax issues that we have yet to discover. At Shopify we are currently taking numerous steps to mitigate this risk, including: * Prism currently passes all Ruby tests and specs, and will continue to going forward. The tests and specs are extensive, and we are confident this gets us almost all of the way there. * Grammar-based fuzzing to ensure we handle all possible syntax variations (both valid and invalid). As an example, that recently led to the discovery that the current compiler was segfaulting on safe navigation in for loops (https://bugs.ruby-lang.org/issues/20468) It has also found several memory leaks that have been fixed. * Mutation testing of existing source files (by adding and deleting random bytes) to find additional edge cases. Our goal is 100% compatibility, and we believe this is achievable through the above techniques and others. The goal of this proposal is to allow more companies to test their codebase against Prism, and to make this easier. We believe that we can sort out the remaining problems much more quickly in that manner. ## Implementation The implementation will be relatively straightforward, as there is already the `--parser` flag that can be used to switch between the current parser and Prism. We will change the default to be `"prism"` instead of `"parse.y"`. There is no plan to remove or change any code related to `parse.y`, `lrama`, or `compile.c` at this time, as we want to create the minimal amount of change possible to ensure stability. -- https://bugs.ruby-lang.org/

4 9

[ruby-core:119413] [Ruby master Bug#20779] Un-rescuable thread creation limit on Mac
by hms (Hal Spitz) 23 Dec '24

23 Dec '24

Issue #20779 has been reported by hms (Hal Spitz). ---------------------------------------- Bug #20779: Un-rescuable thread creation limit on Mac https://bugs.ruby-lang.org/issues/20779 * Author: hms (Hal Spitz) * Status: Open * ruby -v: 3.3.5 * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- Thread.new will hang the ruby process once past the OSX limit. It does not appear that's possible to trap and rescue failed thread creation. It seems the Mac has a limit to the number of threads it supports and googling the issue suggests the limit has been known to change with hardware and OS releases. While I hit the thread creation limit due to a bug in my code and the number of threads that I was attempting to create (12,250+) is an unlikely use-case, I am hoping it would be possible to turn this into a rescuable error. While I'm reporting this on 3.3.5, I have tested the issue and the inability to rescue back to 3.3.1. (Thanks to @raels for being a great second set of eyes to make sure I wasn't doing anything goofy / wrong before making this report. Also, another callout and thanks to both @eregon and @bensheldon over at concurrent-ruby for the unbelievably quick and helpful response to my initial bug report over there). ---Files-------------------------------- hang.rb (713 Bytes) -- https://bugs.ruby-lang.org/

2 1

[ruby-core:115438] [Ruby master Misc#20013] Travis CI status
by jaruga (Jun Aruga) 23 Dec '24

23 Dec '24

Issue #20013 has been reported by jaruga (Jun Aruga). ---------------------------------------- Misc #20013: Travis CI status https://bugs.ruby-lang.org/issues/20013 * Author: jaruga (Jun Aruga) * Status: Open * Priority: Normal ---------------------------------------- I would like to use this ticket to manage our activities to report Travis CI status. Because there is Travis CI status page provided by Travis CI. However, even when the page shows ok, I actually see infra issues. https://www.traviscistatus.com/ I would share my activities and report the Travis CI status on the ticket. The ticket's status is not closed until we stop using Travis CI. The easiest option to fix the Travis infra issue is to email Travis CI support `support _AT_ travis-ci.com`. You can check [this ruby/ruby Travis CI wiki page](https://github.com/ruby/ruby/wiki/CI-Servers#travis-ci) for details. -- https://bugs.ruby-lang.org/

2 49

[ruby-core:117812] [Ruby master Bug#20478] Circular parameter syntax error rules
by kddnewton (Kevin Newton) 20 Dec '24

20 Dec '24

Issue #20478 has been reported by kddnewton (Kevin Newton). ---------------------------------------- Bug #20478: Circular parameter syntax error rules https://bugs.ruby-lang.org/issues/20478 * Author: kddnewton (Kevin Newton) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- I would like to revisit https://bugs.ruby-lang.org/issues/16343. These cases are syntax errors: ```ruby def foo(bar = -> { bar }) end # no lambda parameters def foo(bar = ->() { bar }) end # no lambda parameters def foo(bar = baz { bar }) end # no block parameters def foo(bar = baz { _1 + bar }) end # parameters, but no pipes def foo(bar = baz { it + bar }) end # parameters, but no pipes ``` These cases are not syntax errors: ```ruby def foo(bar = ->(baz) { bar }) end # lambda parameters def foo(bar = baz { || bar }) end # no block parameters but empty pipes def foo(bar = baz { |qux| bar }) end # block parameters ``` I don't think these rules are very intuitive, and they feel somewhat arbitrary. I would like to suggest we change them to be either: * Syntax error is raised if the parameter is ever read in its default value at any scope depth * Syntax error is raised if the parameter is ever read in its default value at depth 0 Either one is fine by me, but gating the syntax error based on the presence of pipes is really confusing. -- https://bugs.ruby-lang.org/

4 10

[ruby-core:119883] [Ruby master Feature#20885] String#gsub?
by Dan0042 (Daniel DeLorme) 18 Dec '24

18 Dec '24

Issue #20885 has been reported by Dan0042 (Daniel DeLorme). ---------------------------------------- Feature #20885: String#gsub? https://bugs.ruby-lang.org/issues/20885 * Author: Dan0042 (Daniel DeLorme) * Status: Open ---------------------------------------- I would like a variation of sub/gsub that returns a new string if there is a match, or nil otherwise. This can be currently done with `str.dup.gsub!(...)` at the mere cost of 2 unnecessary extra allocations due to String#dup TBH I'm not sure we need yet another gsub method, but what gets me here is that `gsub?` is the primitive operation from which `gsub` and `gsub!` are made. Internally, these two methods work like this: ```ruby def gsub!(...) modified = gsub?(...) and replace(modified) end def gsub(...) gsub?(...) or dup end ``` We can efficiently derive these two methods from `gsub?`, but we cannot efficiently implement `gsub?` in ruby, so it feels regrettable to me that `gsub?` is not available as a built-in. Use cases include: ```ruby #ensure gsub has modified the string newstr = str.gsub?(rx,repl) or raise "str did not contain expected value" #take an action if the string is modified if str = obj.title.gsub?(rx,repl) obj.title = str obj.save end ``` -- https://bugs.ruby-lang.org/

2 2

[ruby-core:118957] [Ruby master Bug#20699] On Windows, the `__dir__` keyword is garbled in paths containing Japanese characters, and `require_relative` fails as well
by flatland001 (Hayato Arai) 18 Dec '24

18 Dec '24

Issue #20699 has been reported by flatland001 (Hayato Arai). ---------------------------------------- Bug #20699: On Windows, the `__dir__` keyword is garbled in paths containing Japanese characters, and `require_relative` fails as well https://bugs.ruby-lang.org/issues/20699 * Author: flatland001 (Hayato Arai) * Status: Open * ruby -v: ruby 3.3.4 (2024-07-09 revision be1089c8ec) [x64-mingw-ucrt] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- In paths containing Japanese characters, such as: C:\テスト_test\test.rb Code: ``` ruby # this code -> C:\テスト_test\test.rb # external file -> C:\テスト_test\lib\foo.rb p __dir__ p __dir__.encoding p File.dirname(File.expand_path(__FILE__)) p File.dirname(File.expand_path(__FILE__)).encoding print "__dir__ == File.dirname(File.expand_path(__FILE__)): " p __dir__ == File.dirname(File.expand_path(__FILE__)) begin require_relative "lib/foo" puts "foo.rb loaded" rescue LoadError puts "LoadError: #{$!}" end ``` Results: ``` "C:/?e?X?g_test" #<Encoding:Windows-31J> "C:/\x{8365}\x{8358}\x{8367}_test" #<Encoding:Windows-31J> __dir__ == File.dirname(File.expand_path(__FILE__)): false LoadError: cannot load such file -- C:/?e?X?g_test/lib/foo ``` However, in ruby 3.2.5 (2024-07-26 revision 31d0f1a2e7) [x64-mingw-ucrt], the same code produces the following expected results: ``` "C:/\x{8365}\x{8358}\x{8367}_test" #<Encoding:Windows-31J> "C:/\x{8365}\x{8358}\x{8367}_test" #<Encoding:Windows-31J> __dir__ == File.dirname(File.expand_path(__FILE__)): true foo.rb loaded ``` Is this behavior expected, or could this be a bug in the Ruby interpreter on Windows? If this is my misunderstanding or a known issue, I apologize. Thank you for your consideration. Environment Information: - OS: Windows 11 Pro 23H2 (OS build 22631.4037) - Installed via: RubyInstaller - Shell: Command Prompt - File System: NTFS - System Locale: Japanese (Japan), Language: Japanese -- https://bugs.ruby-lang.org/

3 2

[ruby-core:119380] [Ruby master Misc#20774] Remove remaining locale dependent code from Windows port
by larskanis (Lars Kanis) 18 Dec '24

18 Dec '24

Issue #20774 has been reported by larskanis (Lars Kanis). ---------------------------------------- Misc #20774: Remove remaining locale dependent code from Windows port https://bugs.ruby-lang.org/issues/20774 * Author: larskanis (Lars Kanis) * Status: Open ---------------------------------------- The external_encoding of files, file names and ENV on Windows were changed from locale codepage to UTF-8 in ruby-3.0. But there are still several remaining points where locale encoding is used although there is no need to do so. The Windows port is already fully UTF-16/UTF-8 based and locale encoding is only used for historical and not for technical reasons. My proposal is to remove (most of) the locale dependent conversions from the ruby code for Windows. Before I open pull requests in this regard, I would like to confirm this direction with the ruby core team. Let me show what I mean: ``` # täst-locale-enc.rb def pr(*strs) strs.each do |str| p [str, IO===str ? str.external_encoding&.name : str.encoding.name] end end if $0==__FILE__ pr STDIN # => [#<IO:<STDIN>>, "CP850"] pr $0 # => ["ruby/t\x84st-locale-enc.rb", "CP850"] pr __FILE__ # => ["ruby/t\x84st-locale-enc.rb", "CP850"] pr __dir__ # => ["C:/Users/kanis/ruby", "CP850"] pr 'ä' # => ["ä", "UTF-8"] pr '€' # => ["€", "UTF-8"] pr $:.first # => ["C:/Users/kanis/t\xE2\x82\xACst", "ASCII-8BIT"] pr $:.last # => ["C:/Ruby33-x64/lib/ruby/3.3.0/x64-mingw-ucrt", "CP850"] require "win32/registry" pr Win32::Registry::HKEY_CURRENT_USER.open("Environment")['TMP'] # => ["C:\\Users\\kanis\\AppData\\Local\\Temp", "UTF-8"] pr Win32::Registry::HKEY_CURRENT_USER.open("\\").each_key{ break _1 } # => ["AppEvents", "CP850"] end # execute with: ruby -It€st ruby\täst-locale-enc.rb ``` I wrote the results on `ruby-3.3 x64-mingw-ucrt` right into the code. The situation is even worse when called with `-e` script: ``` $ ruby -It€st -r .\ruby\täst-locale-enc.rb -e "pr STDIN, $0, __FILE__, __dir__, 'ä', '€', $:.first, $:.last" [#<IO:<STDIN>>, "CP850"] ["-e", "CP850"] ["-e", "UTF-8"] [".", "US-ASCII"] ["\x84", "CP850"] ["?", "CP850"] ["C:/Users/kanis/t\xE2\x82\xACst", "ASCII-8BIT"] ["C:/Ruby33-x64/lib/ruby/3.3.0/x64-mingw-ucrt", "CP850"] ``` There are also some inconsistencies like that it's possible to `require` script names with characters outside of the codepage, but it fails to execute a script directly or by using `require_relative` : ``` $ ruby -r .\t€st-locale-enc.rb -e "pr STDIN" [#<IO:<STDIN>>, "CP850"] $ ruby .\t€st-locale-enc.rb ruby: Invalid argument -- ./t?st-locale-enc.rb (LoadError) ``` Maybe there are more places which are working with locale codepage - these are only the few that I remember from memory. I would like to change all the above results to be UTF-8 encoded, like it is the case on Ubuntu. Compatibility ------------- Changing the encoding of returned strings is of course an API change. IMHO it is still something we should change in a minor release of ruby. The reason is that I don't remember about only a single issue cased by the change to UTF-8 in ruby-3.0 in the company I work for. To the contrary many issues are caused by using locale codepage where some non-ASCII characters work and other characters don't. Most issue with ruby-3.0 were cased by the keyword argument changes. -- https://bugs.ruby-lang.org/

2 4

[ruby-core:119925] [Ruby master Feature#20893] random/formatter.rb: Separate from securerandom
by nobu (Nobuyoshi Nakada) 16 Dec '24

16 Dec '24

Issue #20893 has been reported by nobu (Nobuyoshi Nakada). ---------------------------------------- Feature #20893: random/formatter.rb: Separate from securerandom https://bugs.ruby-lang.org/issues/20893 * Author: nobu (Nobuyoshi Nakada) * Status: Open * Assignee: matz (Yukihiro Matsumoto) ---------------------------------------- For historical reasons, random/formatter.rb is included in ruby/securerandom, but it does not contain any security-related methods as evidenced by the fact that its path is not under `securerandom/`, but only contains generic methods for PRNGs, including the standard class `Random`. So I propose to either: 1. put it back in the standard library, or 2. separate it into a new gem. Adding a new dependency may not be welcomed, as seen in the recent issues with base64. -- https://bugs.ruby-lang.org/

2 3

[ruby-core:117657] [Ruby master Bug#20447] Ruby 3.3.1 broken on i686
by vo.x (Vit Ondruch) 15 Dec '24

15 Dec '24

Issue #20447 has been reported by vo.x (Vit Ondruch). ---------------------------------------- Bug #20447: Ruby 3.3.1 broken on i686 https://bugs.ruby-lang.org/issues/20447 * Author: vo.x (Vit Ondruch) * Status: Open * ruby -v: ruby 3.3.1 (2024-04-23 revision c56cd86388) [i686-linux] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- ~~~ gcc -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -march=i686 -mtune=generic -msse2 -mfpmath=sse -mstackrealign -fasynchronous-unwind-tables -fstack-clash-protection -fPIC -m32 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -fno-strict-overflow -fvisibility=hidden -fexcess-precision=standard -DRUBY_EXPORT -I. -I.ext/include/i386-linux -I/builddir/build/BUILD/ruby-3.3.1/include -I/builddir/build/BUILD/ruby-3.3.1 -I/builddir/build/BUILD/ruby-3.3.1/prism -I/builddir/build/BUILD/ruby-3.3.1/enc/unicode/15.0.0 -o time.o -c /builddir/build/BUILD/ruby-3.3.1/time.c make: Leaving directory '/builddir/build/BUILD/ruby-3.3.1/redhat-linux-build' /builddir/build/BUILD/ruby-3.3.1/time.c: In function ‘zone_timelocal’: /builddir/build/BUILD/ruby-3.3.1/time.c:2345:35: error: passing argument 3 of ‘split_second’ from incompatible pointer type [-Wincompatible-pointer-types] 2345 | split_second(tobj->timew, &t, &s); | ^~ | | | wideval_t * {aka long long unsigned int *} /builddir/build/BUILD/ruby-3.3.1/time.c:616:58: note: expected ‘VALUE *’ {aka ‘long unsigned int *’} but argument is of type ‘wideval_t *’ {aka ‘long long unsigned int *’} 616 | split_second(wideval_t timew, wideval_t *timew_p, VALUE *subsecx_p) | ~~~~~~~^~~~~~~~~ make: *** [Makefile:447: time.o] Error 1 make: *** Waiting for unfinished jobs.... make: Entering directory '/builddir/build/BUILD/ruby-3.3.1/redhat-linux-build' ~~~ This is presumably due to #20327 / https://github.com/ruby/ruby/pull/10332. There seems to be medicine in master commit:git|055613fd868a8c94e43893f8c58a00cdd2a81f6d or maybe the whole https://github.com/ruby/ruby/pull/10332, not sure -- https://bugs.ruby-lang.org/

8 14

[ruby-core:119852] [Ruby master Feature#20882] Provide Boolean(...)
by getajobmike (Mike Perham) 14 Dec '24

14 Dec '24

Issue #20882 has been reported by getajobmike (Mike Perham). ---------------------------------------- Feature #20882: Provide Boolean(...) https://bugs.ruby-lang.org/issues/20882 * Author: getajobmike (Mike Perham) * Status: Open ---------------------------------------- Ruby provides Integer(...) and Float(...) global methods to coerce values. Is there a similar method for Booleans? I'd like to do something like: ``` # ENV["SOME_FEATURE"] is unset Boolean(ENV["SOME_FEATURE"]) # => false # ENV["SOME_FEATURE"] is unset, but allow a default? Boolean(ENV["SOME_FEATURE"], true) # => true # explicitly disable ENV["SOME_FEATURE"] = "0" Boolean(ENV["SOME_FEATURE"], true) # => false # explicitly enable ENV["SOME_FEATURE"] = "1" Boolean(ENV["SOME_FEATURE"]) # => true ``` -- https://bugs.ruby-lang.org/

11 11