July 2023 - ruby-core - ml.ruby-lang.org

[ruby-core:113819] [Ruby master Feature#19720] Warning for non-linear Regexps
by Eregon (Benoit Daloze) 14 Sep '23

14 Sep '23

Issue #19720 has been reported by Eregon (Benoit Daloze). ---------------------------------------- Feature #19720: Warning for non-linear Regexps https://bugs.ruby-lang.org/issues/19720 * Author: Eregon (Benoit Daloze) * Status: Open * Priority: Normal ---------------------------------------- I believe the best way to solve ReDoS is to ensure all Regexps used in the process are linear. Using `Regexp.timeout = 5.0` or so does not really prevent ReDoS, given enough requests causing that timeout the servers will still be very unresponsive. To this purpose, we should make it easy to identify non-linear Regexps and fix them. I suggest we either use 1. a performance warning (enabled with `Warning[:performance] = true`, #19538) or 2. a new regexp warning category (enabled with `Warning[:regexp] = true`). I think we should warn only once per non-linear Regexp, to avoid too many such warnings. We could warn as soon as the Regexp is created, or on first match. On first match might makes more sense for Ruby implementations which compile the Regexp lazily (since that is costly during startup), and also avoids warning for Regexps which are never used (which can be good or bad). OTOH, if the warning is enabled, we could always compile the Regexp eagerly (or at least checks whether it's linear), and that would then provide a better way to guarantee that all Regexps created so far are linear. Because warnings are easily customizable, it is also possible to e.g. `raise/abort` on such a warning, if one wants to ensure their application does not use a non-linear Regexp and so cannot be vulnerable to ReDoS: ```ruby Warning.extend Module.new { def warn(message, category: nil, **) raise message if category == :regexp super end } ``` A regexp warning category seems better for that as it makes it easy to filter by category, if a performance warning one would need to match the message which is less clean. As a note, TruffleRuby already has a similar warning, as a command-line option: ``` $ truffleruby --experimental-options --warn-truffle-regex-compile-fallback -e 'Gem' truffleruby-dev/lib/mri/rubygems/version.rb:176: warning: Regexp /\A\s*([0-9]+(?>\.[0-9a-zA-Z]+)*(-[0-9A-Za-z-]+(\.[0-9A-Za-z-]+)*)?)?\s*\z/ at_start=false encoding=US-ASCII requires backtracking and will not match in linear time truffleruby-dev/lib/mri/rubygems/requirement.rb:105: warning: Regexp /\A\s*(=|!=|>|<|>=|<=|~>)?\s*([0-9]+(?>\.[0-9a-zA-Z]+)*(-[0-9A-Za-z-]+(\.[0-9A-Za-z-]+)*)?)\s*\z/ at_start=false encoding=US-ASCII requires backtracking and will not match in linear time ``` So the warning message could be like `FILE:LINE: warning: Regexp /REGEXP/ requires backtracking and might not match in linear time and might cause ReDoS` or more concise: `FILE:LINE: warning: Regexp /REGEXP/ requires backtracking and might cause ReDoS` -- https://bugs.ruby-lang.org/

5 10

[ruby-core:113944] [Ruby master Feature#19737] Add `IO::Buffer#cat` for concat `IO::Buffer` instances
by unasuke (Yusuke Nakamura) 12 Sep '23

12 Sep '23

Issue #19737 has been reported by unasuke (Yusuke Nakamura). ---------------------------------------- Feature #19737: Add `IO::Buffer#cat` for concat `IO::Buffer` instances https://bugs.ruby-lang.org/issues/19737 * Author: unasuke (Yusuke Nakamura) * Status: Open * Priority: Normal ---------------------------------------- ## motivation In my use case, I want to concat two IO::Buffer instances. But current implementation doesn't have that way. Then I created a patch. Opend here: TBD ## concern I have two concerns about it. ### 1. Should we provide `IO::Buffer#+` as an alias? In String instance, `"a" + "b"` returns `"ab",`. It feels intuitive. So, should we provide the same way as `IO::Buffer.for("a") + IO::Buffer.for("b")`? If `+` is provided, I naturally assume that `*` is also provided as an operator. Should we also provide an `IO::Buffer#*` method for symmetry with the String class? I thought the behavior of the "*" method is not obvious to me... (Is it right to just return joined buffers?) ### 2. Should it accept multiple IO::Buffer instances? In the `cat` command, it accepts multiple inputs like this. ``` $ cat a.txt b.txt c.txt a b c ``` Should `IO::Buffer#cat` accept multiple inputs too? -- https://bugs.ruby-lang.org/

4 6

[ruby-core:114319] [Ruby master Feature#19790] Optionally write Ruby crash reports into a file rather than STDERR
by byroot (Jean Boussier) 12 Sep '23

12 Sep '23

Issue #19790 has been reported by byroot (Jean Boussier). ---------------------------------------- Feature #19790: Optionally write Ruby crash reports into a file rather than STDERR https://bugs.ruby-lang.org/issues/19790 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal ---------------------------------------- ### Use case On our servers we set [`/proc/sys/kernel/core_pattern`](https://man7.org/linux/man-pages/man5/core.5.html) to point to a small utility that report all the crashes happening in production with the associated core dump into BugSnag. This allowed us to find and fix many Ruby and native extensions bugs in the last few years. However, these are hard to triage, so we'd like to augment these crash reports with the output of `rb_vm_bugreport()`. ### Problem `rb_vm_bugreport()` is hard coded to print to STDERR, this makes it hard to extract and parse the report in a production environment, as very often STDERR is shared with other processes, so the crash report is intertwined with logs from other processes. ### Feature Request It would be very useful if Ruby could write the crash report to an arbitrary path rather than STDERR, akin to `kernel/core_pattern`. Especially it would be useful if it supported interpolating the crashing process PID with `%p` like `kernel/core_pattern`, as it would make it easier to map that report with the core file. This could be controller by an environment variable such as `RUBY_BUGREPORT_PATH`. e.g. ``` RUBY_BUGREPORT_PATH=/var/log/ruby/ruby-crash-pid-%p.log ``` ### Optional Features `kernel/core_pattern` supports other interpolations, however not all of them would make sense for Ruby to support. %% A single % character. %c Core file size soft resource limit of crashing process (since Linux 2.6.24). %d Dump mode—same as value returned by prctl(2) PR_GET_DUMPABLE (since Linux 3.7). %e The process or thread's comm value, which typically is the same as the executable filename (without path prefix, and truncated to a maximum of 15 characters), but may have been modified to be something different; see the discussion of /proc/pid/comm and /proc/pid/task/tid/comm in proc(5). %E Pathname of executable, with slashes ('/') replaced by exclamation marks ('!') (since Linux 3.0). %g Numeric real GID of dumped process. %h Hostname (same as nodename returned by uname(2)). %i TID of thread that triggered core dump, as seen in the PID namespace in which the thread resides (since Linux 3.18). %I TID of thread that triggered core dump, as seen in the initial PID namespace (since Linux 3.18). %p PID of dumped process, as seen in the PID namespace in which the process resides. %P PID of dumped process, as seen in the initial PID namespace (since Linux 3.12). %s Number of signal causing dump. %t Time of dump, expressed as seconds since the Epoch, 1970-01-01 00:00:00 +0000 (UTC). %u Numeric real UID of dumped process. Additionally, if `kernel/core_pattern` starts with a pipe (`|`), then it allows to pipe the core dump to another program, this may also make sense as a feature. ### Prior Art Aside from `kernel/core_pattern`, some other virtual machine have a similar feature, for instance the JVM has a configurable crash report: ``` -XX:ErrorFile=/var/log/java/hs_err_pid%p.log ``` -- https://bugs.ruby-lang.org/

5 12

[ruby-core:114310] [Ruby master Bug#19788] Ripper returns a symbol instead of a token as operator for "::"
by thyresias (Thierry Lambert) 09 Sep '23

09 Sep '23

Issue #19788 has been reported by thyresias (Thierry Lambert). ---------------------------------------- Bug #19788: Ripper returns a symbol instead of a token as operator for "::" https://bugs.ruby-lang.org/issues/19788 * Author: thyresias (Thierry Lambert) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x64-mingw-ucrt] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- ```ruby require 'ripper' class BasicParser < Ripper EVENTS.each do |event| module_eval(<<~RUBY, __FILE__, __LINE__ + 1) def on_#{event}(*args) puts "#{event}(\#{args.inspect})" args.unshift :#{event} args end RUBY end end require 'pp' BasicParser.new('a&.b').parse # in stdout: # ident(["a"]) # op(["&."]) # vcall([[:ident, "a"]]) # ident(["b"]) # call([[:vcall, [:ident, "a"]], [:op, "&."], [:ident, "b"]]) BasicParser.new('a::b').parse # in stdout: # ident(["a"]) # op(["::"]) # vcall([[:ident, "a"]]) # ident(["b"]) # call([[:vcall, [:ident, "a"]], :"::", [:ident, "b"]]) ``` On the last line, consistent with `&.`, I would have expected: ```ruby # call([[:vcall, [:ident, "a"]], [:op, "::"], [:ident, "b"]]) ``` This is true for the "operator" argument of `call`, `command_call` and `field`. Is it intended? -- https://bugs.ruby-lang.org/

4 3

[ruby-core:111572] [Ruby master Bug#19297] Don't download content from internet to execute Ruby test suite
by vo.x (Vit Ondruch) 06 Sep '23

06 Sep '23

Issue #19297 has been reported by vo.x (Vit Ondruch). ---------------------------------------- Bug #19297: Don't download content from internet to execute Ruby test suite https://bugs.ruby-lang.org/issues/19297 * Author: vo.x (Vit Ondruch) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.0 (2022-12-25 revision a528908271) [x86_64-linux] * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Trying to build Ruby 3.2.0 for Fedora and execute its test suite via `make check` as we always did [1], the test suite suddenly fails (while it was working with commit:git|c5eefb7f37): ~~~ ... snip ... C-API Util function ruby_strtod - converts a string to a double and returns the remaining string - returns 0 and the full string if there's no numerical value Finished in 45.737677 seconds 3827 files, 31635 examples, 177877 expectations, 0 failures, 0 errors, 0 tagged ./miniruby -I/builddir/build/BUILD/ruby-3.2.0/lib -I. -I.ext/common /builddir/build/BUILD/ruby-3.2.0/tool/runruby.rb --extout=.ext -- --disable-gems -C "/builddir/build/BUILD/ruby-3.2.0" bin/gem install --no-document \ --install-dir .bundle --conservative "bundler" "rake" "rspec:~> 3" #"ruby-prof" ERROR: Could not find a valid gem 'bundler' (>= 0), here is why: Unable to download data from https://rubygems.org/ - SocketError: Failed to open TCP connection to rubygems.org:443 (getaddrinfo: Temporary failure in name resolution) (https://rubygems.org/specs.4.8.gz) ERROR: Could not find a valid gem 'rspec' (~> 3), here is why: Unable to download data from https://rubygems.org/ - SocketError: Failed to open TCP connection to rubygems.org:443 (getaddrinfo: Temporary failure in name resolution) (https://rubygems.org/specs.4.8.gz) make: Leaving directory '/builddir/build/BUILD/ruby-3.2.0/redhat-linux-build' make: *** [uncommon.mk:1464: yes-test-syntax-suggest-prepare] Error 2 ~~~ This is obviously due to the test suite trying to download `rspec` from the internet, while Fedora builders does not have internet access (and won't ever have for security reasons). If I am not mistaken, this is caused by commit:git|cae53842735237ccf71a13873fd0d1ae7f165582. Now 1) Can this be fixed? 2) Can the tarball be always self contained? [1]: https://src.fedoraproject.org/rpms/ruby/blob/631163e3b8a51ed610528181aabe0d… -- https://bugs.ruby-lang.org/

8 11

[ruby-core:111499] [Ruby master Bug#19281] SyntaxError if first argument of command call has semicolon inside parenthesis
by tompng (tomoya ishida) 04 Sep '23

04 Sep '23

Issue #19281 has been reported by tompng (tomoya ishida). ---------------------------------------- Bug #19281: SyntaxError if first argument of command call has semicolon inside parenthesis https://bugs.ruby-lang.org/issues/19281 * Author: tompng (tomoya ishida) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- These are syntax error ~~~ruby p (1;2),(3),(4) p (;),(),() a.b (1;2),(3),(4) a.b (;),(),() ~~~ I expect it to be syntax ok because the code below is syntax ok. ~~~ruby p (1),(2;3),(4;5) p (),(;),(;) a.b (1),(2;3),(4;5) a.b (),(;),(;) ~~~ It will be easy to traverse sexp if the sexp of first argument is same as others ~~~ruby Ripper.sexp "p (),(),()" # => [:program, [[:command, [:@ident, "p", [1, 0]], [:args_add_block, [[:paren, false], # [:paren, [[:void_stmt]]] [:paren, [[:void_stmt]]], [:paren, [[:void_stmt]]]], false]]]] Ripper.sexp "p (1),(2),(3)" # => [:program, [[:command, [:@ident, "p", [1, 0]], [:args_add_block, [[:paren, [:@int, "1", [1, 3]]], # [:paren, [[:@int, "1", [1, 3]]]] [:paren, [[:@int, "2", [1, 7]]]], [:paren, [[:@int, "3", [1, 11]]]]], false]]]] ~~~ -- https://bugs.ruby-lang.org/

3 2

[ruby-core:114276] [Ruby master Bug#19784] String#delete_prefix! problem
by inversion (Yura Babak) 01 Sep '23

01 Sep '23

Issue #19784 has been reported by inversion (Yura Babak). ---------------------------------------- Bug #19784: String#delete_prefix! problem https://bugs.ruby-lang.org/issues/19784 * Author: inversion (Yura Babak) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Here is the snipped and the question is in the comments: ``` ruby fp = 'with_BOM_16.txt' body = File.read(fp).force_encoding('UTF-8') p body # "\xFF\xFE1\u00001\u0000" p body.start_with?("\xFF\xFE") # true body.delete_prefix!("\xFF\xFE") # !!! why doesn't work? p body # "\xFF\xFE1\u00001\u0000" p body.start_with?("\xFF\xFE") # true body[0, 2] = '' p body # "1\u00001\u0000" p body.start_with?("\xFF\xFE") # false ``` Works same on Linux (ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux]) and Windows (ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x64-mingw-ucrt]) -- https://bugs.ruby-lang.org/

6 9

[ruby-core:111269] [Ruby master Bug#19230] The openssl backend of securerandom is no longer needed
by mame (Yusuke Endoh) 31 Aug '23

31 Aug '23

Issue #19230 has been reported by mame (Yusuke Endoh). ---------------------------------------- Bug #19230: The openssl backend of securerandom is no longer needed https://bugs.ruby-lang.org/issues/19230 * Author: mame (Yusuke Endoh) * Status: Open * Priority: Normal * ruby -v: ruby 3.1.3p185 (2022-11-24 revision 1a6b16756e) [x86_64-linux] * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN ---------------------------------------- securerandom first checks if Random.urandom is available ([Line 77](https://github.com/ruby/securerandom/blob/5bfe7d6c163f7a8a45af8d2fc377f…) and if not available, it uses the openssl backend as a degeneration. However, the openssl backend does not work because it internally uses Random.urandom ([Line 55](https://github.com/ruby/securerandom/blob/5bfe7d6c163f7a8a45af8d2fc377f…) to create a seed. This issue is found by @hanachin. ``` $ ruby -ve 'def Random.urandom(*); raise; end; require "securerandom"; p SecureRandom.bytes(10)' ruby 3.1.3p185 (2022-11-24 revision 1a6b16756e) [x86_64-linux] -e:1: warning: method redefined; discarding old urandom -e:1:in `urandom': unhandled exception from /home/mame/local/lib/ruby/3.1.0/securerandom.rb:75:in `singleton class' from /home/mame/local/lib/ruby/3.1.0/securerandom.rb:42:in `<module:SecureRandom>' from /home/mame/local/lib/ruby/3.1.0/securerandom.rb:41:in `<top (required)>' from <internal:/home/mame/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require' from <internal:/home/mame/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require' from -e:1:in `<main>' ``` There has been this bug since commit:abae70d6ed63054d7d01bd6cd80c1b5b98b93ba3, which made the urandom backend as default and left the openssl backend just for degeneration. I think no one need the openssl anymore because no one has reported this bug for such a long time. How about removing it? ```diff diff --git a/lib/securerandom.rb b/lib/securerandom.rb index 07ae048634..32b76a2137 100644 --- a/lib/securerandom.rb +++ b/lib/securerandom.rb @@ -14,7 +14,6 @@ # # It supports the following secure random number generators: # -# * openssl # * /dev/urandom # * Win32 # @@ -46,21 +45,6 @@ def bytes(n) private - def gen_random_openssl(n) - @pid = 0 unless defined?(@pid) - pid = $$ - unless @pid == pid - now = Process.clock_gettime(Process::CLOCK_REALTIME, :nanosecond) - OpenSSL::Random.random_add([now, @pid, pid].join(""), 0.0) - seed = Random.urandom(16) - if (seed) - OpenSSL::Random.random_add(seed, 16) - end - @pid = pid - end - return OpenSSL::Random.random_bytes(n) - end - def gen_random_urandom(n) ret = Random.urandom(n) unless ret @@ -77,13 +61,7 @@ def gen_random_urandom(n) Random.urandom(1) alias gen_random gen_random_urandom rescue RuntimeError - begin - require 'openssl' - rescue NoMethodError - raise NotImplementedError, "No random device" - else - alias gen_random gen_random_openssl - end + raise NotImplementedError, "No random device" end public :gen_random ``` -- https://bugs.ruby-lang.org/

6 5

[ruby-core:114248] [Ruby master Feature#19777] Make `Kernel#lambda` raise when called without a literal block
by alanwu (Alan Wu) 24 Aug '23

24 Aug '23

Issue #19777 has been reported by alanwu (Alan Wu). ---------------------------------------- Feature #19777: Make `Kernel#lambda` raise when called without a literal block https://bugs.ruby-lang.org/issues/19777 * Author: alanwu (Alan Wu) * Status: Open * Priority: Normal ---------------------------------------- Since 3.0.0, released in 2020, calling `Kernel#lambda` without a literal block has been issuing a deprecation warning: ```ruby Warning[:deprecated] = true def foo(&b) lambda(&b) end foo {} # => test.rb:2: warning: lambda without a literal block is deprecated; use the proc without lambda instead ``` I think enough time has passed and we should make it raise in all situations where it currently issues a deprecation warning. The original decision to deprecate is here: https://bugs.ruby-lang.org/issues/15973#note-46 The new behavior allows one to predict whether `Kernel#lambda` will return by inspecting its direct caller, checking whether the call site has a literal block. It will remove some hard-to-predict cases where `Kernel#lambda` receives a non-literal block forwarded with `super` or `rb_funcall_passing_block`. The method will always return a lambda, if it returns. However, note that `send` will be a special exception in this new model: ```ruby Warning[:deprecated] = true singleton_class.send(:public, :lambda) p (send(:lambda) {}).lambda? # => true without warning p (public_send(:lambda) {}).lambda? # => true with warning, would raise instead ``` This newer model is friendlier to some optimization we're investigating for YJIT as it has fewer moving parts. -- https://bugs.ruby-lang.org/

4 3

[ruby-core:113954] [Ruby master Misc#19740] Block taking methods can't differentiate between a non-local return and a throw
by byroot (Jean Boussier) 24 Aug '23

24 Aug '23

Issue #19740 has been reported by byroot (Jean Boussier). ---------------------------------------- Misc #19740: Block taking methods can't differentiate between a non-local return and a throw https://bugs.ruby-lang.org/issues/19740 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal ---------------------------------------- Opening this as Misc, as at this stage I don't have a fully formed feature request. Ref: https://github.com/ruby/ruby/commit/1a3bcf103c582b20e9ea70dfed0ee68b24243f55 Ref: https://github.com/ruby/timeout/pull/30 Ref: https://github.com/rails/rails/pull/29333 ### Context Rails has this problem in the Active Record transaction API. The way it works is that it yields to a block, and if no error was raised the SQL transaction is committed, otherwise it's rolled back: ```ruby User.transaction do do_thing end # COMMIT ``` or ```ruby User.transaction do raise SomeError end # ROLLBACK ``` The problem is that there are more ways a method can be exited: ```ruby User.transaction do return # non-local exit end ``` ```ruby User.transaction do throw :something end ``` In the case of a non-local return, we'd want to commit the transaction, but in the case of a throw, particularly since it's internally used by `Timeout.timeout` since Ruby 2.1, we'd rather consider that an error and rollback. But as far as I'm aware, there is not way to distinguish the two cases. ```ruby def transaction returned = false yield returned = true ensure if $! # error was raised elsif returned # no uniwnd else # non-local return or throw, don't know end end ``` I think it could be useful to have a way to access the currently thrown object, similar to `$!` for such cases, or some other way to tell what is going on. There is some discussion going on in https://github.com/ruby/timeout/pull/30 about whether `Timeout` should throw or raise, and that may solve part of the problem, but regardless of where this leads, I think being able to check if something is being thrown would be valuable. cc @matthewd FYI @jeremyevans0 @Eregon -- https://bugs.ruby-lang.org/

3 2