June 2024 - ruby-core - ml.ruby-lang.org

[ruby-core:118275] [Ruby master Feature#19001] Data: Add #to_h symmetric to constructor with keyword args [Follow-on to #16122 Data: simple immutable value object]
by RubyBugs (A Nonymous) 10 Jun '24

10 Jun '24

Issue #19001 has been updated by RubyBugs (A Nonymous). Agreed, this need ended up being met by the initial implementation! Thank you, closing this ticket. zverok (Victor Shepelev) wrote in #note-2: > There isn't any need for this ticket as a separate request, as far as I am concerned. > It works in the initial implementation of data already as submitted in https://bugs.ruby-lang.org/issues/16122#note-68, and even works with ol' good Struct, too: > ```ruby > Point = Struct.new(:x, :y, :z) > > Point[1, 0, 0].to_h.then { Point[**_1] } == Point[1, 0, 0] # => true > ``` ---------------------------------------- Feature #19001: Data: Add #to_h symmetric to constructor with keyword args [Follow-on to #16122 Data: simple immutable value object] https://bugs.ruby-lang.org/issues/19001#change-108768 * Author: RubyBugs (A Nonymous) * Status: Open ---------------------------------------- *Extracted a follow-up to [#16122 Data: simple immutable value object](https://bugs.ruby-lang.org/issues/16122)* # Proposal: Add a `#to_h` method symmetric to a constructor accepting keyword arguments This allows round-trip between a `Hash` and a Value object instance, for example: ```ruby Point = Data.define(:x, :y, :z) points = [ Point.new(x: 1, y: 0, z: 0), Point.new(x: 0, y: 1, z: 0), Point.new(x: 0, y: 0, z: 1), ] hashes = points.map(&:to_h) points_2 = hashes.map { |h| Point.new(**h) } points_2 == points #=> true ``` ## Why? Having symmetric operation between `#to_h` and a keyword-args constructor is a major ergonomic factor in usage of immutable value objects. To play with code that works like this, you may take a look at the [Values gem](https://rubygems.org/gems/values) ## Alternatives If there is no symmetric construction and de-construction along these lines, a number of use cases become more complicated and less ergonomic. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:116039] [Ruby master Bug#20154] aarch64: configure overrides `-mbranch-protection` if it was set in CFLAGS via environment
by jprokop (Jarek Prokop) 09 Jun '24

09 Jun '24

Issue #20154 has been reported by jprokop (Jarek Prokop). ---------------------------------------- Bug #20154: aarch64: configure overrides `-mbranch-protection` if it was set in CFLAGS via environment https://bugs.ruby-lang.org/issues/20154 * Author: jprokop (Jarek Prokop) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- Recently a GH PR was merged <https://github.com/ruby/ruby/pull/9306> For PAC/BTI support on ARM CPUs for Coroutine.S. Without proper compilation support in configure.ac it segfaults Ruby with fibers on CPUs where PAC is supported: https://bugs.ruby-lang.org/issues/20085 At the time of writing, configure.ac appends the first option from a list for flag `-mbranch-protection` that successfully compiles a program <https://github.com/ruby/ruby/blob/master/configure.ac#L829>, to XCFLAGS and now also ASFLAGS to fix issue 20085 for Ruby master. This is suboptimal for Fedora as we set -mbranch-protection=standard by default in C{,XX}FLAGS: ``` CFLAGS='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Werror=implicit-function-declaration -Werror=implicit-int -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -mbranch-protection=standard -fasynchronous-unwind-tables -fstack-clash-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer ' export CFLAGS CXXFLAGS='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -mbranch-protection=standard -fasynchronous-unwind-tables -fstack-clash-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer' export CXXFLAGS ``` And the appended flag overrides distribution's compilation configuration, which in this case ends up omitting BTI instructions and only using PAC. Would it make sense to check if such flags exist and not overwrite them if they do? Serious proposals: 1. Simplest fix that does not overwrite what is set in the distribution and results in higher security is simply prepending the list of options with `-mbranch-protection=standard`, it should cause no problems on ARMv8 CPUs and forward, BTI similarly to PAC instructions result into NOP, it is only extending the capability. See attached 0001-aarch64-Check-mbranch-protection-standard-first-to-u.patch 2. Other fix that sounds more sane IMO and dodges this kind of guessing where are all the correct places for the flag is what another Fedora contributor Florian Weimer suggested: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org… "The reliable way to do this would be to compile a C file and check whether that enables __ARM_FEATURE_PAC_DEFAULT, and if that's the case, define a *different* macro for use in the assembler implementation. This way, you don't need to care about the exact name of the option." IOW instead of using __ARM_FEATURE_* directly in that code, define a macro in the style of "USE_PAC" with value of the feature if it is defined, I think that way we shouldn't need to append ASFLAGS anymore. However it's also important to catch the value of those macros as their values have meaning, I have an idea how to do that but I'd get on that monday earliest. ---Files-------------------------------- 0001-aarch64-Check-mbranch-protection-standard-first-to-u.patch (1004 Bytes) -- https://bugs.ruby-lang.org/

4 10

[ruby-core:118261] [Ruby master Bug#18061] Execshield test: libruby.so.N.N.N: FAIL: property-note test because no .note.gnu.property section found
by kjtsanaktsidis (KJ Tsanaktsidis) 09 Jun '24

09 Jun '24

Issue #18061 has been updated by kjtsanaktsidis (KJ Tsanaktsidis). Assignee set to kjtsanaktsidis (KJ Tsanaktsidis) ---------------------------------------- Bug #18061: Execshield test: libruby.so.N.N.N: FAIL: property-note test because no .note.gnu.property section found https://bugs.ruby-lang.org/issues/18061#change-108759 * Author: jaruga (Jun Aruga) * Status: Open * Assignee: kjtsanaktsidis (KJ Tsanaktsidis) * Backport: 2.6: UNKNOWN, 2.7: REQUIRED, 3.0: REQUIRED ---------------------------------------- I found an issue in our company's internal test called "execshield" by a security tool annobin - annocheck command [1][2]. ``` Hardened: libruby.so.2.7.4: FAIL: property-note test because no .note.gnu.property section found ``` Here is the reproducer on the upstream latest master, commit is 5f2987d6c2ae9ace3178ac3e1bbb4ac7079101eb, ``` $ autoconf $ ./configure --enable-shared $ make $ ls libruby.so.3.1.0 libruby.so.3.1.0* ``` If you are using Red Hat based Linux distro, it's easy to install by the RPM package like this. ``` $ sudo dnf -y install annobin-annocheck ``` ``` $ sudo yum -y install annobin-annocheck ``` Then ``` $ annocheck libruby.so.3.1.0 ``` If you are using other Linux distros such as Ubuntu, you can use it by a container I prepared. Prepare the following `Dockerfile`. ``` $ cat Dockerfile FROM docker.io/fedora:34 RUN cat /etc/fedora-release RUN dnf -y install annobin-annocheck WORKDIR /work ``` Then build the container image with the `Dockerfile` and run the annocheck command for the `libruby.so.3.1.0` on your host environment. The `-v` is an option for bind mount between host and container environment. ``` $ docker build --rm -t fedora-annocheck . $ docker run --rm -t -v $(pwd):/work fedora-annocheck annocheck /work/libruby.so.3.1.0 annocheck: Version 9.79. Hardened: libruby.so.3.1.0: FAIL: bind-now test because not linked with -Wl,-z,now Hardened: libruby.so.3.1.0: FAIL: notes test because gaps were detected in the annobin coverage Hardened: libruby.so.3.1.0: FAIL: cf-protection test because no .note.gnu.property section = no control flow information Hardened: libruby.so.3.1.0: FAIL: property-note test because no .note.gnu.property section found Hardened: Rerun annocheck with --verbose to see more information on the tests. ``` The message `Hardened: libruby.so.3.1.0: FAIL: property-note test because no .note.gnu.property section found` is what I found in our internal test. For other FAIL messages, maybe it can be fixed by changing how to build. Asking a colleague, I was told that the `coroutine/*/Context.S` files such as [coroutine/x86/Context.S](https://github.com/ruby/ruby/blob/master/coroutine… cause the failure. Do you have any idea how to fix this? Thanks. * [1] https://sourceware.org/annobin/ * [2] You can see `man annocheck` or https://www.mankier.com/1/annocheck . ---Files-------------------------------- 0001-Add-.note.gnu.property-sections.patch (2.64 KB) 0001-Add-.note.gnu.property-sections.patch (3.69 KB) config-pie.log (11.4 KB) -- https://bugs.ruby-lang.org/

1 0

[ruby-core:118259] [Ruby master Feature#20567] Net::HTTPResponse#detect_encoding should default to UTF-8 for application/javascript
by postmodern (Hal Brodigan) 09 Jun '24

09 Jun '24

Issue #20567 has been reported by postmodern (Hal Brodigan). ---------------------------------------- Feature #20567: Net::HTTPResponse#detect_encoding should default to UTF-8 for application/javascript https://bugs.ruby-lang.org/issues/20567 * Author: postmodern (Hal Brodigan) * Status: Open ---------------------------------------- It is fairly common now for JavaScript to include UTF-8 characters. If a misconfigured web server returns an `application/javascript` response for a JavaScript file, but *without* `charset=utf-8` in the `Content-Type`, `Net::HTTPResponse` will return a response body with encoding of ASCII-8BIT. This can cause lots of subtle String parsing/encoding issues when the response body is parsed or transformed. `Net::HTTPResponse#detect_encoding` should default the response encoding to UTF-8 if the `Content-Type` includes `application/javascript`. [RFC 4329](https://www.rfc-editor.org/rfc/rfc4329#section-4) also seems to encourage defaulting to UTF-8. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:118257] [Ruby master Bug#20566] string << 0xC2 should raise a RangeError if the string encoding is Encoding::ASCII
by postmodern (Hal Brodigan) 09 Jun '24

09 Jun '24

Issue #20566 has been reported by postmodern (Hal Brodigan). ---------------------------------------- Bug #20566: string << 0xC2 should raise a RangeError if the string encoding is Encoding::ASCII https://bugs.ruby-lang.org/issues/20566 * Author: postmodern (Hal Brodigan) * Status: Open * ruby -v: ruby 3.4.0preview1 (2024-05-16 master 9d69619623) [x86_64-linux] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- I noticed that appending extended ASCII characters (0x80 - 0xff) works when the String's encoding is Encoding::ASCII, as well as Encoding::ASCII_8BIT. I was under the impression that Encoding::ASCII only covers the regular ASCII character set (0x00 - 0x7F). I believe appending 0xC2 to a String with Encoding::ASCII should raise a RangeError. ## Steps To Reproduce ```ruby str = String.new(encoding: Encoding::ASCII) str << 0xC2 ``` ### Expected Result ``` RangeError: character 194 is out of range ``` ### Actual Result ```ruby "\xC2" ``` -- https://bugs.ruby-lang.org/

2 1

[ruby-core:118250] [Ruby master Feature#20565] Relax formal argument at label
by nobu (Nobuyoshi Nakada) 08 Jun '24

08 Jun '24

Issue #20565 has been reported by nobu (Nobuyoshi Nakada). ---------------------------------------- Feature #20565: Relax formal argument at label https://bugs.ruby-lang.org/issues/20565 * Author: nobu (Nobuyoshi Nakada) * Status: Open ---------------------------------------- Just a thought: if `def foo(if:)` is possible, shouldn't `def foo(C:)` and `def foo($g:)` also be possible...? -- https://bugs.ruby-lang.org/

4 3

[ruby-core:118255] [Ruby master Feature#6648] Provide a standard API for retrieving all command-line flags passed to Ruby
by Eregon (Benoit Daloze) 08 Jun '24

08 Jun '24

Issue #6648 has been updated by Eregon (Benoit Daloze). And from a quick look there is also [`_spawnv`](https://learn.microsoft.com/en-us/cpp/c-runtime-library/reference/spawnv-wspawnv?view=msvc-170) which does take a `char**`, maybe we could use that on Windows? ---------------------------------------- Feature #6648: Provide a standard API for retrieving all command-line flags passed to Ruby https://bugs.ruby-lang.org/issues/6648#change-108752 * Author: headius (Charles Nutter) * Status: Assigned * Assignee: matz (Yukihiro Matsumoto) ---------------------------------------- Currently there are no standard mechanisms to get the flags passed to the currently running Ruby implementation. The available mechanisms are not ideal: * Scanning globals and hoping they have not been tweaked to new settings * Using external wrappers to launch Ruby * ??? Inability to get the full set of command-line flags, including flags passed to the VM itself (and probably VM-specific) makes it impossible to launch subprocess Ruby instances with the same settings. A real world example of this is "((%bundle exec%))" when called with a command line that sets various flags, a la ((%jruby -Xsome.vm.setting --1.9 -S bundle exec%)). None of these flags can propagate to the subprocess, so odd behaviors result. The only option is to put the flags into an env var (((|JRUBY_OPTS|)) or ((|RUBYOPT|))) but this breaks the flow of calling a simple command line. JRuby provides mechanisms to get all its command line options, but they require calling Java APIs from Ruby's API set. Rubinius provides its own API for accessing comand-line options, but I do not know if it includes VM-level flags as well as standard Ruby flags. I know there is a (({RubyVM})) namespace in the 2.0 line. If that namespace is intended to be general-purpose for VM-level features, it would be a good host for this API. Something like... ``` class << RubyVM def vm_args; end # returns array of command line args *not* passed to the target script def script; end # returns the script being executed...though this overlaps with $0 def script_args; end # returns args passed to the script...though this overlaps with ARGV, but that is perhaps warranted since ARGV can be modified (i.e. you probably want the original args) end ``` -- https://bugs.ruby-lang.org/

1 0

[ruby-core:118254] [Ruby master Feature#6648] Provide a standard API for retrieving all command-line flags passed to Ruby
by Eregon (Benoit Daloze) 08 Jun '24

08 Jun '24

Issue #6648 has been updated by Eregon (Benoit Daloze). mame (Yusuke Endoh) wrote in #note-23: > [...] instead of parsing the command line to a string array and passing it to `Kernel#exec`. Don't we use `execve()` (which takes `char**`) as well on Windows for `Kernel#exec`? It does seem to exist: https://learn.microsoft.com/en-us/cpp/c-runtime-library/reference/execve-we… Did you mean `Kernel#spawn`/`Kernel#system` maybe? Of course, `RbConfig.ruby_args` would return an Array, not a String. ---------------------------------------- Feature #6648: Provide a standard API for retrieving all command-line flags passed to Ruby https://bugs.ruby-lang.org/issues/6648#change-108751 * Author: headius (Charles Nutter) * Status: Assigned * Assignee: matz (Yukihiro Matsumoto) ---------------------------------------- Currently there are no standard mechanisms to get the flags passed to the currently running Ruby implementation. The available mechanisms are not ideal: * Scanning globals and hoping they have not been tweaked to new settings * Using external wrappers to launch Ruby * ??? Inability to get the full set of command-line flags, including flags passed to the VM itself (and probably VM-specific) makes it impossible to launch subprocess Ruby instances with the same settings. A real world example of this is "((%bundle exec%))" when called with a command line that sets various flags, a la ((%jruby -Xsome.vm.setting --1.9 -S bundle exec%)). None of these flags can propagate to the subprocess, so odd behaviors result. The only option is to put the flags into an env var (((|JRUBY_OPTS|)) or ((|RUBYOPT|))) but this breaks the flow of calling a simple command line. JRuby provides mechanisms to get all its command line options, but they require calling Java APIs from Ruby's API set. Rubinius provides its own API for accessing comand-line options, but I do not know if it includes VM-level flags as well as standard Ruby flags. I know there is a (({RubyVM})) namespace in the 2.0 line. If that namespace is intended to be general-purpose for VM-level features, it would be a good host for this API. Something like... ``` class << RubyVM def vm_args; end # returns array of command line args *not* passed to the target script def script; end # returns the script being executed...though this overlaps with $0 def script_args; end # returns args passed to the script...though this overlaps with ARGV, but that is perhaps warranted since ARGV can be modified (i.e. you probably want the original args) end ``` -- https://bugs.ruby-lang.org/

1 0

[ruby-core:118253] [Ruby master Feature#6648] Provide a standard API for retrieving all command-line flags passed to Ruby
by Eregon (Benoit Daloze) 08 Jun '24

08 Jun '24

Issue #6648 has been updated by Eregon (Benoit Daloze). shyouhei (Shyouhei Urabe) wrote in #note-29: > The current situation is that ruby is not the only valid executable that the method takes. Allowing untrusted inputs for it means it has to be secure for everything. This is too much a hustle. Better find a fine-grained alternative. There is no untrusted input involved here, because the user chooses what flags to pass to `ruby`. If ruby flags can be injected by an attacker, then all is lost regardless of this change (e.g. they can just inject `-r/backdoor.rb`). Regarding the Windows concern, it is the exact same problem for e.g. `spawn("dir", "*.mp3", "/s")`. From what I can see, it is completely separate from this ticket. The code for this on Windows must already escape as much as feasible, and if it fails it's a bug of that code which should be fixed to fix `spawn` etc in general, nothing to change in `RbConfig.ruby_args`. Or if the escaping fails maybe it's just considered a Windows limitation, independent of this ticket. For example, the user is running `ruby --yjit -rmytracing script.rb`, the only addition is the script can now find out the ruby flags it was called with (`["--yjit", "-rmytracing"]`). If the script spawn subprocesses, it already did before, so nothing changes there. It can choose to use `RbConfig.ruby_args`, and that's fine, the user running the script is responsible for whether it's safe to run the script, as always. Let's take the MSpec use-case (mentioned before in https://bugs.ruby-lang.org/issues/6648#note-11) what we want is to run Ruby subprocess with the same Ruby flags. So if e.g. `ruby --yjit -rmytracing path/to/mspec` is called, then if specs create subprocesses (via `ruby_exe()`), then those subprocesses (running some fixture) also have `--yjit -rmytracing`, as desired. You might argue `RUBYOPT` could be used instead, but that is problematic for various reasons: some flags are not allowed in RUBYOPT, RUBYOPT gets propagated arbitrarily far which is not necessarily desired (including to other Ruby implementations and executables written in Ruby, etc). A concrete example I often run into is running ruby/spec with TruffleRuby, I pass `--core-load-path=.../src/main/ruby/truffleruby` to use core library files from disk in development. It is critical that subprocesses in specs also use that (otherwise we'd get an inconsistent core library). The current workaround is to pass that flag both to the ruby process and as `-T`, which is quite ugly but it also slow, because it means mspec must create an extra subprocess just to apply these `-T` flags (it actually uses `exec` but that's just as slow): ``` $ mxbuild/truffleruby-jvm/bin/ruby \ --experimental-options --core-load-path=src/main/ruby/truffleruby \ spec/mspec/bin/mspec run \ --config spec/truffleruby.mspec \ -t .../mxbuild/truffleruby-jvm/bin/ruby \ --excl-tag fails --excl-tag slow \ -T--vm.ea -T--vm.esa \ -T--experimental-options -T--core-load-path=src/main/ruby/truffleruby ``` (as you can see there are already some bugs there because the -T and regular flags don't match exactly). (if you think this could be wrapped in some helper script, it already is, but it changes nothing because it must accept arbitrary ruby flags to be passed) With `RbConfig.ruby_args`, MSpec can know which ruby flags it was passed, which would avoid needing the extra subprocess, and it would be: ``` $ mxbuild/truffleruby-jvm/bin/ruby \ --vm.ea --vm.esa \ --experimental-options --core-load-path=src/main/ruby/truffleruby \ spec/mspec/bin/mspec run \ --config spec/truffleruby.mspec \ -t .../mxbuild/truffleruby-jvm/bin/ruby \ --excl-tag fails --excl-tag slow ``` This would be a killer feature when attaching a debugger, because then one could just `myruby -rmydebug spec/mspec/bin/mspec` and it would start running specs with the debugger (e.g. for TruffleRuby with the Java debugger). Instead of the current situation where the debugger is started on this mspec "wrapper" which just exec's to handle `-T` flags and is very annoying. This happens in CRuby just as much, for example https://github.com/ruby/ruby/blob/69c0b1438a45938e79e63407035f116de4634dcb/… is a workaround causing some duplication. And it makes much more messy to e.g. running a single spec under `gdb`/`lldb`. A very similar situation happens for `make test-all` I would imagine. I guess currently any Ruby subprocesses incorrectly omits Ruby flags, which means the test coverage is lower than intended (e.g. for --yjit, --rjit and all other flags). This would also be convenient for the way to run built-but-not-installed-ruby, which every CRuby developer uses. ---------------------------------------- Feature #6648: Provide a standard API for retrieving all command-line flags passed to Ruby https://bugs.ruby-lang.org/issues/6648#change-108750 * Author: headius (Charles Nutter) * Status: Assigned * Assignee: matz (Yukihiro Matsumoto) ---------------------------------------- Currently there are no standard mechanisms to get the flags passed to the currently running Ruby implementation. The available mechanisms are not ideal: * Scanning globals and hoping they have not been tweaked to new settings * Using external wrappers to launch Ruby * ??? Inability to get the full set of command-line flags, including flags passed to the VM itself (and probably VM-specific) makes it impossible to launch subprocess Ruby instances with the same settings. A real world example of this is "((%bundle exec%))" when called with a command line that sets various flags, a la ((%jruby -Xsome.vm.setting --1.9 -S bundle exec%)). None of these flags can propagate to the subprocess, so odd behaviors result. The only option is to put the flags into an env var (((|JRUBY_OPTS|)) or ((|RUBYOPT|))) but this breaks the flow of calling a simple command line. JRuby provides mechanisms to get all its command line options, but they require calling Java APIs from Ruby's API set. Rubinius provides its own API for accessing comand-line options, but I do not know if it includes VM-level flags as well as standard Ruby flags. I know there is a (({RubyVM})) namespace in the 2.0 line. If that namespace is intended to be general-purpose for VM-level features, it would be a good host for this API. Something like... ``` class << RubyVM def vm_args; end # returns array of command line args *not* passed to the target script def script; end # returns the script being executed...though this overlaps with $0 def script_args; end # returns args passed to the script...though this overlaps with ARGV, but that is perhaps warranted since ARGV can be modified (i.e. you probably want the original args) end ``` -- https://bugs.ruby-lang.org/

1 0

[ruby-core:118147] [Ruby master Misc#20519] Porting regexp to pure ruby?
by brightbits (Michael Baldry) 07 Jun '24

07 Jun '24

Issue #20519 has been reported by brightbits (Michael Baldry). ---------------------------------------- Misc #20519: Porting regexp to pure ruby? https://bugs.ruby-lang.org/issues/20519 * Author: brightbits (Michael Baldry) * Status: Open ---------------------------------------- Would there be any benefit in porting Regexp from Onigmo to a pure ruby implementation that could benefit from YJIT? Compiling a pattern could be translating to a ruby method which would be optimized by YJIT easily. Has this been explored or any work done around this kind of thing, before I take a look in to it more? Many thanks -- https://bugs.ruby-lang.org/

3 4