July 2023 - ruby-core - ml.ruby-lang.org

[ruby-core:114072] [Ruby master Bug#19754] `IO::Buffer#get_string` raises unsuitable exception for too large offset
by nobu (Nobuyoshi Nakada) 06 Nov '23

06 Nov '23

Issue #19754 has been reported by nobu (Nobuyoshi Nakada). ---------------------------------------- Bug #19754: `IO::Buffer#get_string` raises unsuitable exception for too large offset https://bugs.ruby-lang.org/issues/19754 * Author: nobu (Nobuyoshi Nakada) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- The message seems suitable when both of `offset` and `length` are given. ```ruby IO::Buffer.for("test").get_string(5, 0) #=> Specified offset+length exceeds buffer size! (ArgumentError) ``` But without `length`, it doesn't seem to check `offset` properly. ```ruby IO::Buffer.for("test").get_string(5) #=> negative string size (or size too big) (ArgumentError) ``` -- https://bugs.ruby-lang.org/

4 6

[ruby-core:111953] [Ruby master Bug#19362] #dup on Proc doesn't call initialize_dup
by zverok (Victor Shepelev) 26 Oct '23

26 Oct '23

Issue #19362 has been reported by zverok (Victor Shepelev). ---------------------------------------- Bug #19362: #dup on Proc doesn't call initialize_dup https://bugs.ruby-lang.org/issues/19362 * Author: zverok (Victor Shepelev) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- In #17545, `#dup` had changed to create an instance of the subclass. It, though, doesn't invoke `initialize_dup` of the subclass, unlike other standard classes. ```ruby class MyAry < Array def initialize_dup(...) p(self.class, ...) super end end class MyString < String def initialize_dup(...) p(self.class, ...) super end end class MyProc < Proc def initialize_dup(...) p(self.class, ...) super end end MyString.new('test').dup # prints MyString, "test" MyAry.new(['test']).dup # prints MyAry, ["test"] MyProc.new { 'test' }.dup # doesn't print anything ``` This makes the change in #17545 useless: while inheriting from core classes is indeed marginal, one of author's intention might be carrying additional information with the Proc instance, and bypassing `#initialize_dup` makes it impossible to maintain this information. It seems that actually `#initialize_dup` is also invoked on the core classes themselves, but ignored on `Proc`. ```ruby class Array def initialize_dup(...) p(self.class, ...) super end end class String def initialize_dup(...) p(self.class, ...) super end end class Proc def initialize_dup(...) p(self.class, ...) super end end 'test'.dup # prints String, "test" ['test'].dup # prints Array, ["test"] Proc.new { 'test' }.dup # doesn't print anything ``` Which is an even more marginal problem but still an inconsistency. -- https://bugs.ruby-lang.org/

3 7

[ruby-core:112944] [Ruby master Feature#19541] Proposal: Generate frame unwinding info for YJIT code
by kjtsanaktsidis (KJ Tsanaktsidis) 19 Oct '23

19 Oct '23

Issue #19541 has been reported by kjtsanaktsidis (KJ Tsanaktsidis). ---------------------------------------- Feature #19541: Proposal: Generate frame unwinding info for YJIT code https://bugs.ruby-lang.org/issues/19541 * Author: kjtsanaktsidis (KJ Tsanaktsidis) * Status: Open * Priority: Normal ---------------------------------------- ## What is being propsed? Currently, Ruby crashes with yjit generated code on the stack, `rb_print_backtrace()` is unable to actually show any frames underneath the yjit code. For example, if you send SIGSEGV to a Ruby process running yjit, this is what you see: ``` /ruby/miniruby(rb_print_backtrace+0xc) [0xaaaad0276884] /ruby/vm_dump.c:785 /ruby/miniruby(rb_vm_bugreport) /ruby/vm_dump.c:1093 /ruby/miniruby(rb_bug_for_fatal_signal+0xd0) [0xaaaad0075580] /ruby/error.c:813 /ruby/miniruby(sigsegv+0x5c) [0xaaaad01bedac] /ruby/signal.c:919 linux-vdso.so.1(__kernel_rt_sigreturn+0x0) [0xffff91a3e8bc] /ruby/miniruby(map<(usize, yjit::backend::ir::Insn), (usize, yjit::backend::ir::Insn), yjit::backend::ir::{impl#17}::next_mapped::{closure_env#0}>+0x8c) [0xaaaad03b8b00] /rustc/897e37553bba8b42751c67658967889d11ecd120/library/core/src/option.rs:929 /ruby/miniruby(next_mapped+0x3c) [0xaaaad0291dc0] src/backend/ir.rs:1225 /ruby/miniruby(arm64_split+0x114) [0xaaaad0287744] src/backend/arm64/mod.rs:359 /ruby/miniruby(compile_with_regs+0x80) [0xaaaad028bf84] src/backend/arm64/mod.rs:1106 /ruby/miniruby(compile+0xc4) [0xaaaad0291ae0] src/backend/ir.rs:1158 /ruby/miniruby(gen_single_block+0xe44) [0xaaaad02b1f88] src/codegen.rs:854 /ruby/miniruby(gen_block_series_body+0x9c) [0xaaaad03b0250] src/core.rs:1698 /ruby/miniruby(gen_block_series+0x50) [0xaaaad03b0100] src/core.rs:1676 /ruby/miniruby(branch_stub_hit_body+0x80c) [0xaaaad03b1f68] src/core.rs:2021 /ruby/miniruby({closure#0}+0x28) [0xaaaad02eb86c] src/core.rs:1924 /ruby/miniruby(do_call<yjit::core::branch_stub_hit::{closure_env#0}, *const u8>+0x98) [0xaaaad035ba3c] /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/panicking.rs:492 [0xaaaad035c9b4] ``` (n.b. - I compiled Ruby with `-fasynchronous-unwind-tables –rdynamic –g` in cflags to make sure gcc generates appropriate unwind info & keeps the symbol tables). Likewise, if you attach gdb to a Ruby process with yjit enabled, gdb can't show thread backtraces through yjit-generated code either. My proposal is that YJIT generate sufficient unwinding and debug information on all platforms to allow both `rb_print_backtrace()` and the platform's debugger (gdb/lldb/WinDbg) to show: * Full stack traces all the way back to `main`. That is, it should be possible to see frames _underneath_ `[0xaaaad035c9b4]` from the backtrace above. * Names for the dynamically generated yjit blocks (e.g. instead of `[0xaaaad035c9b4]`, we should see something like `yjit$$name_of_ruby_method`, where `name_of_ruby_method` is the `label` for the iseq this is JIT'd code for). ## Motivation I have a few motivations for wanting this. Firstly, I feel this functionality is independently useful. When Ruby crashes, the more information we can get, the more likely we are to find the root cause. Likewise, the same principle applies to debugging with gdb - you can get a fuller understanding of what the process is doing if you see the whole stack. I have often found attaching gdb to the Ruby interpreter helps in understanding problems in Ruby code or C extensions and is something I do relatively frequently; yjit breaking that will definitely be inconvenient for me! ## Implementation I have a draft implementation here on how I'd implement this: . It's currently missing tests & platform support (it only works on Linux aarch64). Also, it implements unwind info generation, so unwinding can work _through_ yjit code, but it does not currently emit symbols to give _names_ to those yjit frames. My PR contains a document which explains how the Linux interfaces for registering unwind info for JIT'd code work, so I won't duplicate that information here. The biggest implementation question I had is around the use of Rust crates. Currently, I prototyped my implementation using the gimli & object crates, for generating DWARF info and ELF binaries. However, the yjit build does purposefully does not use cargo & external crates for release builds. There are a few different ways we could go here: * Don't use the gimli & object crates; instead, re-implement all debug info & object file generation code in yjit. * Don't use the crates; instead, link againt C libraries to provide this functionality & call them from Rust (perhaps some combination of libelf, libdw, libbfd, or llvm might do what we need) * Use cargo after all for the release build & download the crates at build-time * Use cargo for the release build, but vendor everything, so the build doesn't need to download anything * Only make unwind info generation available in dev mode where cargo is used, and so mark the gimli/object dependencies as optional in Cargo.toml. We'd need to decide on one of these approaches for this proposal to work. I don't really have a strong sense of the pros/cons of each. (Side note - my PR actually depends on a _fork_ of gimli - I've been discussing adding the needed interfaces upstream here: https://github.com/gimli-rs/gimli/issues/648) ## Benchmarks I ran the yit-bench suite on my branch and compared it to Ruby master: * My branch: https://gist.github.com/KJTsanaktsidis/5741a9f64e5cd75cdf5fedd846091a4f * Ruby master: https://gist.github.com/KJTsanaktsidis/592d3ebcf98f6745dfa3efbd30a25acf This is a (very simple) comparison: ``` -------------- ------------ ------------ --------------- bench yjit (ms) branch (ms) branch/yjit (%) activerecord 97.5 98.5 101.03% hexapdf 2415.3 2458.2 101.78% liquid-c 61.9 63.1 101.94% liquid-render 135.3 135.0 99.78% mail 104.6 105.5 100.86% psych-load 1887.1 1922.0 101.85% railsbench 1544.4 1556.0 100.75% ruby-lsp 88.4 89.5 101.24% sequel 147.5 151.1 102.44% binarytrees 303 305.6 100.86% chunky_png 1075.8 1079.4 100.33% erubi 392.9 392.3 99.85% erubi_rails 14.7 14.7 100.00% etanni 792.3 791.4 99.89% fannkuchredux 3815.9 3813.6 99.94% lee 1030.2 1039.2 100.87% nbody 49.2 49.3 100.20% optcarrot 4142 4143.3 100.03% ruby-json 2860.7 2874.0 100.46% rubykon 7906.6 7904.2 99.97% 30k_ifelse 348.7 345.4 99.05% 30k_methods 828.6 831.8 100.39% cfunc_itself 28.8 28.9 100.35% fib 34.4 34.5 100.29% getivar 115.5 109.7 94.98% keyword_args 37.7 38.0 100.80% respond_to 26 26.1 100.38% setivar 33.8 33.5 99.11% setivar_object 208.7 194.3 93.10% str_concat 52.6 52.2 99.24% throw 23.8 24.1 101.26% -------------- ------------ ------------ --------------- ``` It seems like the performance impact of generating and registering the debug info is marginal. -- https://bugs.ruby-lang.org/

7 12

[ruby-core:114250] [Ruby master Bug#19778] mkmf.rb pkg_config() interaction with RbConfig::CONFIG["cflags"]
by rhenium (Kazuki Yamaguchi) 17 Oct '23

17 Oct '23

Issue #19778 has been reported by rhenium (Kazuki Yamaguchi). ---------------------------------------- Bug #19778: mkmf.rb pkg_config() interaction with RbConfig::CONFIG["cflags"] https://bugs.ruby-lang.org/issues/19778 * Author: rhenium (Kazuki Yamaguchi) * Status: Open * Priority: Normal * ruby -v: ruby 3.3.0dev (2023-07-21T09:38:29Z master 22f9735587) [x86_64-linux] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- (This was first reported at https://github.com/ruby/openssl/issues/650: The extension's build breaks in a strange way if `RbConfig::CONFIG["*flags"]` contains the path of an OpenSSL installation but `pkg-config` returns the path of a different OpenSSL installation) Commit commit:097c3e9cbbf23718371f08c24b2d2297b039f63f ("mkmf.rb: -I flags to $INCFLAGS", Ruby 2.2) changed how mkmf's `pkg_config()` handles the result of the `pkg-config` command. It now stores the `-I` flags in $INCFLAGS and others in $CFLAGS. mkmf generates a Makefile that compiles source files with: $(CC) $(INCFLAGS) $(CPPFLAGS) $(CFLAGS) and link the final library with: $(LDSHARED) -o $@ $(OBJS) $(LIBPATH) $(DLDFLAGS) $(LOCAL_LIBS) $(LIBS) This "new" behavior of `pkg_config()` is problematic when `RbConfig::CONFIG["{C,CPP}FLAGS"]` also provide `-I` flags and `RbConfig::CONFIG["LDFLAGS"]` provides the matching `-L` flags -- for example, if Ruby is compiled with `./configure --with-opt-dir=<dir>`. This would end up with compiling sources with [...] -I<from pkg-config> -I<from RbConfig> [...] and then linking with [...] -L<from RbConfig> -L<from pkg-config> [...] This doesn't seem right. I don't know which should come earlier, but the order should be consistent. The commit in question clearly describes the change in the commit message, but it doesn't have a linked issue. What is it intended for? Also, what is $INCFLAGS? On the other hand, `dir_config()` would prepend `-I` flags to $CPPFLAGS and `-L` flags to $LIBPATH, so it doesn't have issues with flags from `RbConfig`, albeit in a differently way. -- https://bugs.ruby-lang.org/

5 4

[ruby-core:114181] [Ruby master Bug#19767] [Not really a bug, but more a not ideal notification] "historical binary regexp match" when using the "n" modifier in a ruby regex
by rubyFeedback (robert heiler) 29 Sep '23

29 Sep '23

Issue #19767 has been reported by rubyFeedback (robert heiler). ---------------------------------------- Bug #19767: [Not really a bug, but more a not ideal notification] "historical binary regexp match" when using the "n" modifier in a ruby regex https://bugs.ruby-lang.org/issues/19767 * Author: rubyFeedback (robert heiler) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- To get my knowledge about ruby regexes up-to-date I have been going through this tutorial/book here at: https://learnbyexample.github.io/Ruby_Regexp/unicode.html One example they provide is this, with some odd characters: 'fox:αλεπού'.scan(/\w+/n) This will match the found word ("fox"), but it also reports the following warning: warning: historical binary regexp match /.../n against UTF-8 string Now: this may be obvious to others, but to me personally I am not sure what a "historical" binary regexp match actually is. I assume it may have meant that this was more used in the past, and may be discouraged now? Or is something else meant? What does "historical" mean in this context? I may not be the only one who does not fully understand the term historical. Most of ruby's warnings are fairly easy to understand, but this one seems odd. Right now I do not know whether we can use the "n" modifier in a regex - not that I really have a good use case for it (I am using UTF-8 these days, so I don't seem to need ASCII-8BIT anyway), but perhaps the warning could be changed a little. I have no good alternative suggestion how it can be changed, largely because I do not know what it actually means, e. g. what is "historical" about it (but, even then, I'd actually recommend against using the word "historical" because I don't understand what it means; deprecated is easy to understand, historical does not tell me anything). Perhaps it could be expressed somewhat differently and we could get rid of the word "historical" there? Either way, it's a tiny issue so I was not even sure whether to report it. But, from the point of view of other warnings, I believe the term "historical" does not tell the user enough about what the issue is here. (irb):1: warning: historical binary regexp match /.../n against UTF-8 string => ["fox"] -- https://bugs.ruby-lang.org/

3 2

[ruby-core:114222] [Ruby master Misc#19772] API Naming for YARP compiler
by jemmai (Jemma Issroff) 22 Sep '23

22 Sep '23

Issue #19772 has been reported by jemmai (Jemma Issroff). ---------------------------------------- Misc #19772: API Naming for YARP compiler https://bugs.ruby-lang.org/issues/19772 * Author: jemmai (Jemma Issroff) * Status: Open * Priority: Normal ---------------------------------------- We are working on the YARP compiler, and have [the first PR ready](https://github.com/ruby/ruby/pull/8042) which introduces the YARP compile method. Our only outstanding question before merging it is about naming. How should we expose the public API for YARP's compile method? Potential suggestions: 1. YARP.compile 2. RubyVM::InstructionSequence.compile(yarp: true) 3. RubyVM::InstructionSequence.compile_yarp 4. Any of the above options, with a name other than yarp (please suggest an alternative) Regarding option 1, which would mirror `YARP.parse`, is the top level constant `YARP` acceptable? cc @matz @ko1 -- https://bugs.ruby-lang.org/

9 30

[ruby-core:113381] [Ruby master Bug#19624] Backticks - IO object leakage
by pineman 20 Sep '23

20 Sep '23

Issue #19624 has been reported by pineman (João Pinheiro). ---------------------------------------- Bug #19624: Backticks - IO object leakage https://bugs.ruby-lang.org/issues/19624 * Author: pineman (João Pinheiro) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.2 (2023-03-30 revision e51014f9c0) [arm64-darwin22] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Hi, This code works on ruby 3.0.6: ```ruby `echo` ObjectSpace.each_object(IO) do |io| if ![STDIN, STDOUT, STDERR].include?(io) io.close end end ``` but raises `IOError` on 3.2.2: ``` minimal-repro-case.rb:8:in `close': uninitialized stream (IOError) ``` I found it started failing on ruby 3.1.0 and after, on macOS and Linux. This code is useful for closing unneeded IO objects in forked processes. It looks like backticks is 'leaking' IO objects, waiting for GC, and it didn't used to in 3.1.0. In ruby 3.1.0, inside `rb_f_backquote`, `rb_gc_force_recycle` was removed in favor of `RB_GC_GUARD`. I wonder if this has something to do with the problem. Is this code incorrect since ruby 3.1.0 or is it a bug in ruby? Thanks. ---Files-------------------------------- minimal-repro-case.rb (109 Bytes) -- https://bugs.ruby-lang.org/

4 9

[ruby-core:113926] [Ruby master Bug#19735] Add support for UUID version 7
by nevans (Nicholas Evans) 19 Sep '23

19 Sep '23

Issue #19735 has been reported by nevans (Nicholas Evans). ---------------------------------------- Bug #19735: Add support for UUID version 7 https://bugs.ruby-lang.org/issues/19735 * Author: nevans (Nicholas Evans) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Although the specification for UUIDv7 is still in draft, the UUIDv7 algorithm has been stable as the RFC progresses to completion. Version 7 UUIDs can be very useful, because they are lexographically sortable, which can improve e.g: database index locality. See section 6.10 of the draft specification for further explanation: https://www.ietf.org/archive/id/draft-ietf-uuidrev-rfc4122bis-06.html ```ruby require 'random/formatter' Random.uuid_v7 # => "0188ca50-fcc0-7881-b5c5-6d55cd8fc373" Random.uuid_v7 # => "0188ca51-0069-7304-be2e-0c3cd908789b" Random.uuid_v7 # => "0188ca51-04aa-7b57-a6ec-c49573412a9d" Random.uuid_v7 # => "0188ca51-0853-7979-ae37-485460e9f4f1" # or prng = Random.new prng.uuid_v7 # => "0188ca51-5e72-7950-a11d-def7ff977c98" ``` PR here: https://github.com/ruby/ruby/pull/7953 -- https://bugs.ruby-lang.org/

3 4

[ruby-core:113819] [Ruby master Feature#19720] Warning for non-linear Regexps
by Eregon (Benoit Daloze) 14 Sep '23

14 Sep '23

Issue #19720 has been reported by Eregon (Benoit Daloze). ---------------------------------------- Feature #19720: Warning for non-linear Regexps https://bugs.ruby-lang.org/issues/19720 * Author: Eregon (Benoit Daloze) * Status: Open * Priority: Normal ---------------------------------------- I believe the best way to solve ReDoS is to ensure all Regexps used in the process are linear. Using `Regexp.timeout = 5.0` or so does not really prevent ReDoS, given enough requests causing that timeout the servers will still be very unresponsive. To this purpose, we should make it easy to identify non-linear Regexps and fix them. I suggest we either use 1. a performance warning (enabled with `Warning[:performance] = true`, #19538) or 2. a new regexp warning category (enabled with `Warning[:regexp] = true`). I think we should warn only once per non-linear Regexp, to avoid too many such warnings. We could warn as soon as the Regexp is created, or on first match. On first match might makes more sense for Ruby implementations which compile the Regexp lazily (since that is costly during startup), and also avoids warning for Regexps which are never used (which can be good or bad). OTOH, if the warning is enabled, we could always compile the Regexp eagerly (or at least checks whether it's linear), and that would then provide a better way to guarantee that all Regexps created so far are linear. Because warnings are easily customizable, it is also possible to e.g. `raise/abort` on such a warning, if one wants to ensure their application does not use a non-linear Regexp and so cannot be vulnerable to ReDoS: ```ruby Warning.extend Module.new { def warn(message, category: nil, **) raise message if category == :regexp super end } ``` A regexp warning category seems better for that as it makes it easy to filter by category, if a performance warning one would need to match the message which is less clean. As a note, TruffleRuby already has a similar warning, as a command-line option: ``` $ truffleruby --experimental-options --warn-truffle-regex-compile-fallback -e 'Gem' truffleruby-dev/lib/mri/rubygems/version.rb:176: warning: Regexp /\A\s*([0-9]+(?>\.[0-9a-zA-Z]+)*(-[0-9A-Za-z-]+(\.[0-9A-Za-z-]+)*)?)?\s*\z/ at_start=false encoding=US-ASCII requires backtracking and will not match in linear time truffleruby-dev/lib/mri/rubygems/requirement.rb:105: warning: Regexp /\A\s*(=|!=|>|<|>=|<=|~>)?\s*([0-9]+(?>\.[0-9a-zA-Z]+)*(-[0-9A-Za-z-]+(\.[0-9A-Za-z-]+)*)?)\s*\z/ at_start=false encoding=US-ASCII requires backtracking and will not match in linear time ``` So the warning message could be like `FILE:LINE: warning: Regexp /REGEXP/ requires backtracking and might not match in linear time and might cause ReDoS` or more concise: `FILE:LINE: warning: Regexp /REGEXP/ requires backtracking and might cause ReDoS` -- https://bugs.ruby-lang.org/

5 10

[ruby-core:113944] [Ruby master Feature#19737] Add `IO::Buffer#cat` for concat `IO::Buffer` instances
by unasuke (Yusuke Nakamura) 12 Sep '23

12 Sep '23

Issue #19737 has been reported by unasuke (Yusuke Nakamura). ---------------------------------------- Feature #19737: Add `IO::Buffer#cat` for concat `IO::Buffer` instances https://bugs.ruby-lang.org/issues/19737 * Author: unasuke (Yusuke Nakamura) * Status: Open * Priority: Normal ---------------------------------------- ## motivation In my use case, I want to concat two IO::Buffer instances. But current implementation doesn't have that way. Then I created a patch. Opend here: TBD ## concern I have two concerns about it. ### 1. Should we provide `IO::Buffer#+` as an alias? In String instance, `"a" + "b"` returns `"ab",`. It feels intuitive. So, should we provide the same way as `IO::Buffer.for("a") + IO::Buffer.for("b")`? If `+` is provided, I naturally assume that `*` is also provided as an operator. Should we also provide an `IO::Buffer#*` method for symmetry with the String class? I thought the behavior of the "*" method is not obvious to me... (Is it right to just return joined buffers?) ### 2. Should it accept multiple IO::Buffer instances? In the `cat` command, it accepts multiple inputs like this. ``` $ cat a.txt b.txt c.txt a b c ``` Should `IO::Buffer#cat` accept multiple inputs too? -- https://bugs.ruby-lang.org/

4 6