July 2023 - ruby-core - ml.ruby-lang.org

[ruby-core:114222] [Ruby master Misc#19772] API Naming for YARP compiler
by jemmai (Jemma Issroff) 22 Sep '23

22 Sep '23

Issue #19772 has been reported by jemmai (Jemma Issroff). ---------------------------------------- Misc #19772: API Naming for YARP compiler https://bugs.ruby-lang.org/issues/19772 * Author: jemmai (Jemma Issroff) * Status: Open * Priority: Normal ---------------------------------------- We are working on the YARP compiler, and have [the first PR ready](https://github.com/ruby/ruby/pull/8042) which introduces the YARP compile method. Our only outstanding question before merging it is about naming. How should we expose the public API for YARP's compile method? Potential suggestions: 1. YARP.compile 2. RubyVM::InstructionSequence.compile(yarp: true) 3. RubyVM::InstructionSequence.compile_yarp 4. Any of the above options, with a name other than yarp (please suggest an alternative) Regarding option 1, which would mirror `YARP.parse`, is the top level constant `YARP` acceptable? cc @matz @ko1 -- https://bugs.ruby-lang.org/

9 30

[ruby-core:113381] [Ruby master Bug#19624] Backticks - IO object leakage
by pineman 20 Sep '23

20 Sep '23

Issue #19624 has been reported by pineman (João Pinheiro). ---------------------------------------- Bug #19624: Backticks - IO object leakage https://bugs.ruby-lang.org/issues/19624 * Author: pineman (João Pinheiro) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.2 (2023-03-30 revision e51014f9c0) [arm64-darwin22] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Hi, This code works on ruby 3.0.6: ```ruby `echo` ObjectSpace.each_object(IO) do |io| if ![STDIN, STDOUT, STDERR].include?(io) io.close end end ``` but raises `IOError` on 3.2.2: ``` minimal-repro-case.rb:8:in `close': uninitialized stream (IOError) ``` I found it started failing on ruby 3.1.0 and after, on macOS and Linux. This code is useful for closing unneeded IO objects in forked processes. It looks like backticks is 'leaking' IO objects, waiting for GC, and it didn't used to in 3.1.0. In ruby 3.1.0, inside `rb_f_backquote`, `rb_gc_force_recycle` was removed in favor of `RB_GC_GUARD`. I wonder if this has something to do with the problem. Is this code incorrect since ruby 3.1.0 or is it a bug in ruby? Thanks. ---Files-------------------------------- minimal-repro-case.rb (109 Bytes) -- https://bugs.ruby-lang.org/

4 9

[ruby-core:113926] [Ruby master Bug#19735] Add support for UUID version 7
by nevans (Nicholas Evans) 19 Sep '23

19 Sep '23

Issue #19735 has been reported by nevans (Nicholas Evans). ---------------------------------------- Bug #19735: Add support for UUID version 7 https://bugs.ruby-lang.org/issues/19735 * Author: nevans (Nicholas Evans) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Although the specification for UUIDv7 is still in draft, the UUIDv7 algorithm has been stable as the RFC progresses to completion. Version 7 UUIDs can be very useful, because they are lexographically sortable, which can improve e.g: database index locality. See section 6.10 of the draft specification for further explanation: https://www.ietf.org/archive/id/draft-ietf-uuidrev-rfc4122bis-06.html ```ruby require 'random/formatter' Random.uuid_v7 # => "0188ca50-fcc0-7881-b5c5-6d55cd8fc373" Random.uuid_v7 # => "0188ca51-0069-7304-be2e-0c3cd908789b" Random.uuid_v7 # => "0188ca51-04aa-7b57-a6ec-c49573412a9d" Random.uuid_v7 # => "0188ca51-0853-7979-ae37-485460e9f4f1" # or prng = Random.new prng.uuid_v7 # => "0188ca51-5e72-7950-a11d-def7ff977c98" ``` PR here: https://github.com/ruby/ruby/pull/7953 -- https://bugs.ruby-lang.org/

3 4

[ruby-core:113819] [Ruby master Feature#19720] Warning for non-linear Regexps
by Eregon (Benoit Daloze) 14 Sep '23

14 Sep '23

Issue #19720 has been reported by Eregon (Benoit Daloze). ---------------------------------------- Feature #19720: Warning for non-linear Regexps https://bugs.ruby-lang.org/issues/19720 * Author: Eregon (Benoit Daloze) * Status: Open * Priority: Normal ---------------------------------------- I believe the best way to solve ReDoS is to ensure all Regexps used in the process are linear. Using `Regexp.timeout = 5.0` or so does not really prevent ReDoS, given enough requests causing that timeout the servers will still be very unresponsive. To this purpose, we should make it easy to identify non-linear Regexps and fix them. I suggest we either use 1. a performance warning (enabled with `Warning[:performance] = true`, #19538) or 2. a new regexp warning category (enabled with `Warning[:regexp] = true`). I think we should warn only once per non-linear Regexp, to avoid too many such warnings. We could warn as soon as the Regexp is created, or on first match. On first match might makes more sense for Ruby implementations which compile the Regexp lazily (since that is costly during startup), and also avoids warning for Regexps which are never used (which can be good or bad). OTOH, if the warning is enabled, we could always compile the Regexp eagerly (or at least checks whether it's linear), and that would then provide a better way to guarantee that all Regexps created so far are linear. Because warnings are easily customizable, it is also possible to e.g. `raise/abort` on such a warning, if one wants to ensure their application does not use a non-linear Regexp and so cannot be vulnerable to ReDoS: ```ruby Warning.extend Module.new { def warn(message, category: nil, **) raise message if category == :regexp super end } ``` A regexp warning category seems better for that as it makes it easy to filter by category, if a performance warning one would need to match the message which is less clean. As a note, TruffleRuby already has a similar warning, as a command-line option: ``` $ truffleruby --experimental-options --warn-truffle-regex-compile-fallback -e 'Gem' truffleruby-dev/lib/mri/rubygems/version.rb:176: warning: Regexp /\A\s*([0-9]+(?>\.[0-9a-zA-Z]+)*(-[0-9A-Za-z-]+(\.[0-9A-Za-z-]+)*)?)?\s*\z/ at_start=false encoding=US-ASCII requires backtracking and will not match in linear time truffleruby-dev/lib/mri/rubygems/requirement.rb:105: warning: Regexp /\A\s*(=|!=|>|<|>=|<=|~>)?\s*([0-9]+(?>\.[0-9a-zA-Z]+)*(-[0-9A-Za-z-]+(\.[0-9A-Za-z-]+)*)?)\s*\z/ at_start=false encoding=US-ASCII requires backtracking and will not match in linear time ``` So the warning message could be like `FILE:LINE: warning: Regexp /REGEXP/ requires backtracking and might not match in linear time and might cause ReDoS` or more concise: `FILE:LINE: warning: Regexp /REGEXP/ requires backtracking and might cause ReDoS` -- https://bugs.ruby-lang.org/

5 10

[ruby-core:113944] [Ruby master Feature#19737] Add `IO::Buffer#cat` for concat `IO::Buffer` instances
by unasuke (Yusuke Nakamura) 12 Sep '23

12 Sep '23

Issue #19737 has been reported by unasuke (Yusuke Nakamura). ---------------------------------------- Feature #19737: Add `IO::Buffer#cat` for concat `IO::Buffer` instances https://bugs.ruby-lang.org/issues/19737 * Author: unasuke (Yusuke Nakamura) * Status: Open * Priority: Normal ---------------------------------------- ## motivation In my use case, I want to concat two IO::Buffer instances. But current implementation doesn't have that way. Then I created a patch. Opend here: TBD ## concern I have two concerns about it. ### 1. Should we provide `IO::Buffer#+` as an alias? In String instance, `"a" + "b"` returns `"ab",`. It feels intuitive. So, should we provide the same way as `IO::Buffer.for("a") + IO::Buffer.for("b")`? If `+` is provided, I naturally assume that `*` is also provided as an operator. Should we also provide an `IO::Buffer#*` method for symmetry with the String class? I thought the behavior of the "*" method is not obvious to me... (Is it right to just return joined buffers?) ### 2. Should it accept multiple IO::Buffer instances? In the `cat` command, it accepts multiple inputs like this. ``` $ cat a.txt b.txt c.txt a b c ``` Should `IO::Buffer#cat` accept multiple inputs too? -- https://bugs.ruby-lang.org/

4 6

[ruby-core:114319] [Ruby master Feature#19790] Optionally write Ruby crash reports into a file rather than STDERR
by byroot (Jean Boussier) 12 Sep '23

12 Sep '23

Issue #19790 has been reported by byroot (Jean Boussier). ---------------------------------------- Feature #19790: Optionally write Ruby crash reports into a file rather than STDERR https://bugs.ruby-lang.org/issues/19790 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal ---------------------------------------- ### Use case On our servers we set [`/proc/sys/kernel/core_pattern`](https://man7.org/linux/man-pages/man5/core.5.html) to point to a small utility that report all the crashes happening in production with the associated core dump into BugSnag. This allowed us to find and fix many Ruby and native extensions bugs in the last few years. However, these are hard to triage, so we'd like to augment these crash reports with the output of `rb_vm_bugreport()`. ### Problem `rb_vm_bugreport()` is hard coded to print to STDERR, this makes it hard to extract and parse the report in a production environment, as very often STDERR is shared with other processes, so the crash report is intertwined with logs from other processes. ### Feature Request It would be very useful if Ruby could write the crash report to an arbitrary path rather than STDERR, akin to `kernel/core_pattern`. Especially it would be useful if it supported interpolating the crashing process PID with `%p` like `kernel/core_pattern`, as it would make it easier to map that report with the core file. This could be controller by an environment variable such as `RUBY_BUGREPORT_PATH`. e.g. ``` RUBY_BUGREPORT_PATH=/var/log/ruby/ruby-crash-pid-%p.log ``` ### Optional Features `kernel/core_pattern` supports other interpolations, however not all of them would make sense for Ruby to support. %% A single % character. %c Core file size soft resource limit of crashing process (since Linux 2.6.24). %d Dump mode—same as value returned by prctl(2) PR_GET_DUMPABLE (since Linux 3.7). %e The process or thread's comm value, which typically is the same as the executable filename (without path prefix, and truncated to a maximum of 15 characters), but may have been modified to be something different; see the discussion of /proc/pid/comm and /proc/pid/task/tid/comm in proc(5). %E Pathname of executable, with slashes ('/') replaced by exclamation marks ('!') (since Linux 3.0). %g Numeric real GID of dumped process. %h Hostname (same as nodename returned by uname(2)). %i TID of thread that triggered core dump, as seen in the PID namespace in which the thread resides (since Linux 3.18). %I TID of thread that triggered core dump, as seen in the initial PID namespace (since Linux 3.18). %p PID of dumped process, as seen in the PID namespace in which the process resides. %P PID of dumped process, as seen in the initial PID namespace (since Linux 3.12). %s Number of signal causing dump. %t Time of dump, expressed as seconds since the Epoch, 1970-01-01 00:00:00 +0000 (UTC). %u Numeric real UID of dumped process. Additionally, if `kernel/core_pattern` starts with a pipe (`|`), then it allows to pipe the core dump to another program, this may also make sense as a feature. ### Prior Art Aside from `kernel/core_pattern`, some other virtual machine have a similar feature, for instance the JVM has a configurable crash report: ``` -XX:ErrorFile=/var/log/java/hs_err_pid%p.log ``` -- https://bugs.ruby-lang.org/

5 12

[ruby-core:114310] [Ruby master Bug#19788] Ripper returns a symbol instead of a token as operator for "::"
by thyresias (Thierry Lambert) 09 Sep '23

09 Sep '23

Issue #19788 has been reported by thyresias (Thierry Lambert). ---------------------------------------- Bug #19788: Ripper returns a symbol instead of a token as operator for "::" https://bugs.ruby-lang.org/issues/19788 * Author: thyresias (Thierry Lambert) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x64-mingw-ucrt] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- ```ruby require 'ripper' class BasicParser < Ripper EVENTS.each do |event| module_eval(<<~RUBY, __FILE__, __LINE__ + 1) def on_#{event}(*args) puts "#{event}(\#{args.inspect})" args.unshift :#{event} args end RUBY end end require 'pp' BasicParser.new('a&.b').parse # in stdout: # ident(["a"]) # op(["&."]) # vcall([[:ident, "a"]]) # ident(["b"]) # call([[:vcall, [:ident, "a"]], [:op, "&."], [:ident, "b"]]) BasicParser.new('a::b').parse # in stdout: # ident(["a"]) # op(["::"]) # vcall([[:ident, "a"]]) # ident(["b"]) # call([[:vcall, [:ident, "a"]], :"::", [:ident, "b"]]) ``` On the last line, consistent with `&.`, I would have expected: ```ruby # call([[:vcall, [:ident, "a"]], [:op, "::"], [:ident, "b"]]) ``` This is true for the "operator" argument of `call`, `command_call` and `field`. Is it intended? -- https://bugs.ruby-lang.org/

4 3

[ruby-core:111572] [Ruby master Bug#19297] Don't download content from internet to execute Ruby test suite
by vo.x (Vit Ondruch) 06 Sep '23

06 Sep '23

Issue #19297 has been reported by vo.x (Vit Ondruch). ---------------------------------------- Bug #19297: Don't download content from internet to execute Ruby test suite https://bugs.ruby-lang.org/issues/19297 * Author: vo.x (Vit Ondruch) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.0 (2022-12-25 revision a528908271) [x86_64-linux] * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Trying to build Ruby 3.2.0 for Fedora and execute its test suite via `make check` as we always did [1], the test suite suddenly fails (while it was working with commit:git|c5eefb7f37): ~~~ ... snip ... C-API Util function ruby_strtod - converts a string to a double and returns the remaining string - returns 0 and the full string if there's no numerical value Finished in 45.737677 seconds 3827 files, 31635 examples, 177877 expectations, 0 failures, 0 errors, 0 tagged ./miniruby -I/builddir/build/BUILD/ruby-3.2.0/lib -I. -I.ext/common /builddir/build/BUILD/ruby-3.2.0/tool/runruby.rb --extout=.ext -- --disable-gems -C "/builddir/build/BUILD/ruby-3.2.0" bin/gem install --no-document \ --install-dir .bundle --conservative "bundler" "rake" "rspec:~> 3" #"ruby-prof" ERROR: Could not find a valid gem 'bundler' (>= 0), here is why: Unable to download data from https://rubygems.org/ - SocketError: Failed to open TCP connection to rubygems.org:443 (getaddrinfo: Temporary failure in name resolution) (https://rubygems.org/specs.4.8.gz) ERROR: Could not find a valid gem 'rspec' (~> 3), here is why: Unable to download data from https://rubygems.org/ - SocketError: Failed to open TCP connection to rubygems.org:443 (getaddrinfo: Temporary failure in name resolution) (https://rubygems.org/specs.4.8.gz) make: Leaving directory '/builddir/build/BUILD/ruby-3.2.0/redhat-linux-build' make: *** [uncommon.mk:1464: yes-test-syntax-suggest-prepare] Error 2 ~~~ This is obviously due to the test suite trying to download `rspec` from the internet, while Fedora builders does not have internet access (and won't ever have for security reasons). If I am not mistaken, this is caused by commit:git|cae53842735237ccf71a13873fd0d1ae7f165582. Now 1) Can this be fixed? 2) Can the tarball be always self contained? [1]: https://src.fedoraproject.org/rpms/ruby/blob/631163e3b8a51ed610528181aabe0d… -- https://bugs.ruby-lang.org/

8 11

[ruby-core:111499] [Ruby master Bug#19281] SyntaxError if first argument of command call has semicolon inside parenthesis
by tompng (tomoya ishida) 04 Sep '23

04 Sep '23

Issue #19281 has been reported by tompng (tomoya ishida). ---------------------------------------- Bug #19281: SyntaxError if first argument of command call has semicolon inside parenthesis https://bugs.ruby-lang.org/issues/19281 * Author: tompng (tomoya ishida) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- These are syntax error ~~~ruby p (1;2),(3),(4) p (;),(),() a.b (1;2),(3),(4) a.b (;),(),() ~~~ I expect it to be syntax ok because the code below is syntax ok. ~~~ruby p (1),(2;3),(4;5) p (),(;),(;) a.b (1),(2;3),(4;5) a.b (),(;),(;) ~~~ It will be easy to traverse sexp if the sexp of first argument is same as others ~~~ruby Ripper.sexp "p (),(),()" # => [:program, [[:command, [:@ident, "p", [1, 0]], [:args_add_block, [[:paren, false], # [:paren, [[:void_stmt]]] [:paren, [[:void_stmt]]], [:paren, [[:void_stmt]]]], false]]]] Ripper.sexp "p (1),(2),(3)" # => [:program, [[:command, [:@ident, "p", [1, 0]], [:args_add_block, [[:paren, [:@int, "1", [1, 3]]], # [:paren, [[:@int, "1", [1, 3]]]] [:paren, [[:@int, "2", [1, 7]]]], [:paren, [[:@int, "3", [1, 11]]]]], false]]]] ~~~ -- https://bugs.ruby-lang.org/

3 2

[ruby-core:114276] [Ruby master Bug#19784] String#delete_prefix! problem
by inversion (Yura Babak) 01 Sep '23

01 Sep '23

Issue #19784 has been reported by inversion (Yura Babak). ---------------------------------------- Bug #19784: String#delete_prefix! problem https://bugs.ruby-lang.org/issues/19784 * Author: inversion (Yura Babak) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Here is the snipped and the question is in the comments: ``` ruby fp = 'with_BOM_16.txt' body = File.read(fp).force_encoding('UTF-8') p body # "\xFF\xFE1\u00001\u0000" p body.start_with?("\xFF\xFE") # true body.delete_prefix!("\xFF\xFE") # !!! why doesn't work? p body # "\xFF\xFE1\u00001\u0000" p body.start_with?("\xFF\xFE") # true body[0, 2] = '' p body # "1\u00001\u0000" p body.start_with?("\xFF\xFE") # false ``` Works same on Linux (ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux]) and Windows (ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x64-mingw-ucrt]) -- https://bugs.ruby-lang.org/

6 9