June 2024 - ruby-core - ml.ruby-lang.org

[ruby-core:118302] [Ruby master Bug#20577] FREE_AT_EXIT segfaulting after 32683aa1
by mdalessio (Mike Dalessio) 12 Jun '24

12 Jun '24

Issue #20577 has been reported by mdalessio (Mike Dalessio). ---------------------------------------- Bug #20577: FREE_AT_EXIT segfaulting after 32683aa1 https://bugs.ruby-lang.org/issues/20577 * Author: mdalessio (Mike Dalessio) * Status: Open * ruby -v: ruby 3.4.0dev (2024-06-11T14:33:51Z master 32683aa18d) [x86_64-linux] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- After commit 32683aa1 on master, using `RUBY_FREE_AT_EXIT` results in a segfault. When run with valgrind, many invalid reads are reported. Repro with ``` RUBY_FREE_AT_EXIT=1 bundle exec /path/to/ruby -e 'puts RUBY_DESCRIPTION' ``` or for valgrind details ``` RUBY_FREE_AT_EXIT=1 bundle exec valgrind /path/to/ruby -e 'puts RUBY_DESCRIPTION' ``` Typical example valgrind error is: ``` ==441955== Invalid read of size 4 ==441955== at 0x2C6F97: hash_table_index (id_table.c:132) ==441955== by 0x2C6F97: rb_id_table_lookup (id_table.c:230) ==441955== by 0x31AB50: cached_callable_method_entry (vm_method.c:1415) ==441955== by 0x31AB50: callable_method_entry_or_negative (vm_method.c:1485) ==441955== by 0x31C4EE: callable_method_entry (vm_method.c:1522) ==441955== by 0x31C4EE: rb_callable_method_entry (vm_method.c:1529) ==441955== by 0x18F370: obj_any_hash (hash.c:222) ==441955== by 0x19F444: any_hash (hash.c:204) ==441955== by 0x19F444: rb_any_hash (hash.c:260) ==441955== by 0x19F444: ar_do_hash (hash.c:418) ==441955== by 0x19F444: ar_delete (hash.c:1093) ==441955== by 0x19F444: rb_hash_stlike_delete (hash.c:2385) ==441955== by 0x1A0CCB: rb_hash_delete_entry (hash.c:2402) ==441955== by 0x2C59B6: rb_gc_free_dsymbol (symbol.c:851) ==441955== by 0x17AC53: obj_free (gc.c:3460) ==441955== by 0x17D045: rb_objspace_call_finalizer_i (gc.c:4352) ==441955== by 0x17D045: gc_each_object (gc.c:4302) ==441955== by 0x17D045: rb_objspace_call_finalizer (gc.c:4401) ==441955== by 0x162719: rb_ec_finalize (eval.c:164) ==441955== by 0x162719: rb_ec_cleanup (eval.c:255) ==441955== by 0x15CE09: rb_main (main.c:40) ==441955== by 0x15CE09: main (main.c:59) ==441955== Address 0x4ec4610 is 0 bytes inside a block of size 24 free'd ==441955== at 0x484988F: free (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so) ==441955== by 0x175BD1: objspace_xfree (gc.c:12153) ==441955== by 0x175BD1: objspace_xfree (gc.c:12084) ==441955== by 0x175BD1: ruby_sized_xfree.constprop.0 (gc.c:12257) ==441955== by 0x17A9A0: obj_free (gc.c:3293) ==441955== by 0x17D045: rb_objspace_call_finalizer_i (gc.c:4352) ==441955== by 0x17D045: gc_each_object (gc.c:4302) ==441955== by 0x17D045: rb_objspace_call_finalizer (gc.c:4401) ==441955== by 0x162719: rb_ec_finalize (eval.c:164) ==441955== by 0x162719: rb_ec_cleanup (eval.c:255) ==441955== by 0x15CE09: rb_main (main.c:40) ==441955== by 0x15CE09: main (main.c:59) ==441955== Block was alloc'd at ==441955== at 0x4846828: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so) ==441955== by 0x189B0A: objspace_xmalloc0 (gc.c:11938) ==441955== by 0x2C6EA6: rb_id_table_create (id_table.c:98) ==441955== by 0x31AD99: cache_callable_method_entry (vm_method.c:1444) ==441955== by 0x31AD99: callable_method_entry_or_negative (vm_method.c:1502) ==441955== by 0x32A7DE: callable_method_entry (vm_method.c:1522) ==441955== by 0x32A7DE: vm_respond_to (vm_method.c:2878) ==441955== by 0x3305E3: check_funcall_respond_to (vm_eval.c:602) ==441955== by 0x3305E3: rb_check_funcall_default_kw (vm_eval.c:676) ==441955== by 0x20E896: convert_type_with_id (object.c:3131) ==441955== by 0x20E896: rb_check_convert_type_with_id (object.c:3223) ==441955== by 0x1672A7: rb_stat (file.c:1299) ==441955== by 0x16E15E: rb_file_exist_p (file.c:1805) ==441955== by 0x16E15E: rb_check_realpath_internal (file.c:4567) ==441955== by 0x1D6639: get_loaded_features_index (load.c:415) ==441955== by 0x1D6FB7: rb_provide_feature (load.c:703) ==441955== by 0x1A2629: rb_call_inits (inits.c:64) ==441955== by 0x162B67: ruby_setup (eval.c:86) ==441955== by 0x162C1C: ruby_init (eval.c:98) ==441955== by 0x15CDF7: rb_main (main.c:39) ==441955== by 0x15CDF7: main (main.c:59) ``` I'm attaching a full log from my system. cc @peterzhu2118 ---Files-------------------------------- bug.log (238 KB) -- https://bugs.ruby-lang.org/

2 5

[ruby-core:116790] [Ruby master Bug#20270] Options with `--parser=prism`
by nobu (Nobuyoshi Nakada) 12 Jun '24

12 Jun '24

Issue #20270 has been reported by nobu (Nobuyoshi Nakada). ---------------------------------------- Bug #20270: Options with `--parser=prism` https://bugs.ruby-lang.org/issues/20270 * Author: nobu (Nobuyoshi Nakada) * Status: Open * Priority: Normal * Backport: 3.0: DONTNEED, 3.1: DONTNEED, 3.2: DONTNEED, 3.3: REQUIRED ---------------------------------------- ### `--dump` option Currently `parsetree` and `prism_parsetree` bits are separated, but it seems meaningless as far as `--parser` option selects only one parser. Why doesn't simply `--dump=parsetree` dump AST by parse.y, or PRISM AST if `--parser=prism` is given? Actually, `ruby --dump=parsetree --parser=prism` just segfaults, because these bits and `--parser` option are separately handled. ### streaming code from stdin ```sh-session $ echo end | ruby --parser=prism - ruby: warning: The compiler based on the Prism parser is currently experimental and compatibility with the compiler based on parse.y is not yet complete. Please report any issues you find on the `ruby/prism` issue tracker. -: warning: Prism support for streaming code from stdin is not currently supported ``` Nothing reported is fine, since it is not currently supported. I was a bit curious that it looks like trying something after the warning. ```C if (strcmp(opt->script, "-") == 0) { int xflag = opt->xflag; VALUE rb_source = open_load_file(opt->script_name, &xflag); opt->xflag = xflag != 0; rb_warn("Prism support for streaming code from stdin is not currently supported"); error = pm_parse_string(&result, rb_source, opt->script_name); } ``` Note that `open_load_file` returns `rb_stdin` when `-` is given. This object is a `RFile`, not a `RString`, and `RSTRING_PTR(rb_source)` accesses after the object boundary. It should just bail out, not try obviously wrong thing. ### Fix https://github.com/nobu/ruby/tree/options-refactor -- https://bugs.ruby-lang.org/

2 3

[ruby-core:118154] [Ruby master Bug#20521] Memory leak in Ripper parsing
by kddnewton (Kevin Newton) 11 Jun '24

11 Jun '24

Issue #20521 has been reported by kddnewton (Kevin Newton). ---------------------------------------- Bug #20521: Memory leak in Ripper parsing https://bugs.ruby-lang.org/issues/20521 * Author: kddnewton (Kevin Newton) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- Currently, this code is leaking: ```ruby require "ripper" source = <<~'RUBY' <<-A; %w[j\ i A j] RUBY 10.times do 100_000.times do Ripper.sexp_raw(source) end puts `ps -o rss= -p #{$$}` end ``` This results in: ``` 31584 36304 41504 47008 51712 56384 61088 65792 70480 75200 ``` I haven't managed to track it down, but I can see that 3.2.2 does not leak but 3.3.0 does leak. -- https://bugs.ruby-lang.org/

3 2

[ruby-core:118173] [Ruby master Misc#20524] mswin and gmp
by MSP-Greg (Greg L) 11 Jun '24

11 Jun '24

Issue #20524 has been reported by MSP-Greg (Greg L). ---------------------------------------- Misc #20524: mswin and gmp https://bugs.ruby-lang.org/issues/20524 * Author: MSP-Greg (Greg L) * Status: Open ---------------------------------------- Currently, the `vcpkg.json` file exists, and it can be used to specify what MSFT/vcpkg packages to install, and also a repo commit, which will determine the what package versions are used. The file lists the gmp package. 1. To use the gmp package in mswin, `win32/configure.bat` shows an option of `--with-gmp`. This option is not used in the `windows.yml` workflow file, nor is it mentioned in `doc/windows.md`. 2. When using the option, the build fails when compiling `bignum.c`, as it can’t find `gmp.h`. It appears the vcpkg include folder is not being passed to `cl`? So, it seems a few things are amiss. `gmp` was added a long time ago. With current mswin builds, will `gmp` improve `Bignum`’s performance? Is that unknown? If there are benchmarks to determine this, I could certainly run them… -- https://bugs.ruby-lang.org/

3 2

[ruby-core:118273] [Ruby master Bug#20572] Compilation hangs with nested until
by kddnewton (Kevin Newton) 11 Jun '24

11 Jun '24

Issue #20572 has been reported by kddnewton (Kevin Newton). ---------------------------------------- Bug #20572: Compilation hangs with nested until https://bugs.ruby-lang.org/issues/20572 * Author: kddnewton (Kevin Newton) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- ```ruby until while 1 end end ``` this code causes an infinite loop in the compiler. -- https://bugs.ruby-lang.org/

2 2

[ruby-core:118275] [Ruby master Feature#19001] Data: Add #to_h symmetric to constructor with keyword args [Follow-on to #16122 Data: simple immutable value object]
by RubyBugs (A Nonymous) 10 Jun '24

10 Jun '24

Issue #19001 has been updated by RubyBugs (A Nonymous). Agreed, this need ended up being met by the initial implementation! Thank you, closing this ticket. zverok (Victor Shepelev) wrote in #note-2: > There isn't any need for this ticket as a separate request, as far as I am concerned. > It works in the initial implementation of data already as submitted in https://bugs.ruby-lang.org/issues/16122#note-68, and even works with ol' good Struct, too: > ```ruby > Point = Struct.new(:x, :y, :z) > > Point[1, 0, 0].to_h.then { Point[**_1] } == Point[1, 0, 0] # => true > ``` ---------------------------------------- Feature #19001: Data: Add #to_h symmetric to constructor with keyword args [Follow-on to #16122 Data: simple immutable value object] https://bugs.ruby-lang.org/issues/19001#change-108768 * Author: RubyBugs (A Nonymous) * Status: Open ---------------------------------------- *Extracted a follow-up to [#16122 Data: simple immutable value object](https://bugs.ruby-lang.org/issues/16122)* # Proposal: Add a `#to_h` method symmetric to a constructor accepting keyword arguments This allows round-trip between a `Hash` and a Value object instance, for example: ```ruby Point = Data.define(:x, :y, :z) points = [ Point.new(x: 1, y: 0, z: 0), Point.new(x: 0, y: 1, z: 0), Point.new(x: 0, y: 0, z: 1), ] hashes = points.map(&:to_h) points_2 = hashes.map { |h| Point.new(**h) } points_2 == points #=> true ``` ## Why? Having symmetric operation between `#to_h` and a keyword-args constructor is a major ergonomic factor in usage of immutable value objects. To play with code that works like this, you may take a look at the [Values gem](https://rubygems.org/gems/values) ## Alternatives If there is no symmetric construction and de-construction along these lines, a number of use cases become more complicated and less ergonomic. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:116039] [Ruby master Bug#20154] aarch64: configure overrides `-mbranch-protection` if it was set in CFLAGS via environment
by jprokop (Jarek Prokop) 10 Jun '24

10 Jun '24

Issue #20154 has been reported by jprokop (Jarek Prokop). ---------------------------------------- Bug #20154: aarch64: configure overrides `-mbranch-protection` if it was set in CFLAGS via environment https://bugs.ruby-lang.org/issues/20154 * Author: jprokop (Jarek Prokop) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- Recently a GH PR was merged <https://github.com/ruby/ruby/pull/9306> For PAC/BTI support on ARM CPUs for Coroutine.S. Without proper compilation support in configure.ac it segfaults Ruby with fibers on CPUs where PAC is supported: https://bugs.ruby-lang.org/issues/20085 At the time of writing, configure.ac appends the first option from a list for flag `-mbranch-protection` that successfully compiles a program <https://github.com/ruby/ruby/blob/master/configure.ac#L829>, to XCFLAGS and now also ASFLAGS to fix issue 20085 for Ruby master. This is suboptimal for Fedora as we set -mbranch-protection=standard by default in C{,XX}FLAGS: ``` CFLAGS='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Werror=implicit-function-declaration -Werror=implicit-int -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -mbranch-protection=standard -fasynchronous-unwind-tables -fstack-clash-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer ' export CFLAGS CXXFLAGS='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -mbranch-protection=standard -fasynchronous-unwind-tables -fstack-clash-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer' export CXXFLAGS ``` And the appended flag overrides distribution's compilation configuration, which in this case ends up omitting BTI instructions and only using PAC. Would it make sense to check if such flags exist and not overwrite them if they do? Serious proposals: 1. Simplest fix that does not overwrite what is set in the distribution and results in higher security is simply prepending the list of options with `-mbranch-protection=standard`, it should cause no problems on ARMv8 CPUs and forward, BTI similarly to PAC instructions result into NOP, it is only extending the capability. See attached 0001-aarch64-Check-mbranch-protection-standard-first-to-u.patch 2. Other fix that sounds more sane IMO and dodges this kind of guessing where are all the correct places for the flag is what another Fedora contributor Florian Weimer suggested: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org… "The reliable way to do this would be to compile a C file and check whether that enables __ARM_FEATURE_PAC_DEFAULT, and if that's the case, define a *different* macro for use in the assembler implementation. This way, you don't need to care about the exact name of the option." IOW instead of using __ARM_FEATURE_* directly in that code, define a macro in the style of "USE_PAC" with value of the feature if it is defined, I think that way we shouldn't need to append ASFLAGS anymore. However it's also important to catch the value of those macros as their values have meaning, I have an idea how to do that but I'd get on that monday earliest. ---Files-------------------------------- 0001-aarch64-Check-mbranch-protection-standard-first-to-u.patch (1004 Bytes) -- https://bugs.ruby-lang.org/

4 10

[ruby-core:118261] [Ruby master Bug#18061] Execshield test: libruby.so.N.N.N: FAIL: property-note test because no .note.gnu.property section found
by kjtsanaktsidis (KJ Tsanaktsidis) 09 Jun '24

09 Jun '24

Issue #18061 has been updated by kjtsanaktsidis (KJ Tsanaktsidis). Assignee set to kjtsanaktsidis (KJ Tsanaktsidis) ---------------------------------------- Bug #18061: Execshield test: libruby.so.N.N.N: FAIL: property-note test because no .note.gnu.property section found https://bugs.ruby-lang.org/issues/18061#change-108759 * Author: jaruga (Jun Aruga) * Status: Open * Assignee: kjtsanaktsidis (KJ Tsanaktsidis) * Backport: 2.6: UNKNOWN, 2.7: REQUIRED, 3.0: REQUIRED ---------------------------------------- I found an issue in our company's internal test called "execshield" by a security tool annobin - annocheck command [1][2]. ``` Hardened: libruby.so.2.7.4: FAIL: property-note test because no .note.gnu.property section found ``` Here is the reproducer on the upstream latest master, commit is 5f2987d6c2ae9ace3178ac3e1bbb4ac7079101eb, ``` $ autoconf $ ./configure --enable-shared $ make $ ls libruby.so.3.1.0 libruby.so.3.1.0* ``` If you are using Red Hat based Linux distro, it's easy to install by the RPM package like this. ``` $ sudo dnf -y install annobin-annocheck ``` ``` $ sudo yum -y install annobin-annocheck ``` Then ``` $ annocheck libruby.so.3.1.0 ``` If you are using other Linux distros such as Ubuntu, you can use it by a container I prepared. Prepare the following `Dockerfile`. ``` $ cat Dockerfile FROM docker.io/fedora:34 RUN cat /etc/fedora-release RUN dnf -y install annobin-annocheck WORKDIR /work ``` Then build the container image with the `Dockerfile` and run the annocheck command for the `libruby.so.3.1.0` on your host environment. The `-v` is an option for bind mount between host and container environment. ``` $ docker build --rm -t fedora-annocheck . $ docker run --rm -t -v $(pwd):/work fedora-annocheck annocheck /work/libruby.so.3.1.0 annocheck: Version 9.79. Hardened: libruby.so.3.1.0: FAIL: bind-now test because not linked with -Wl,-z,now Hardened: libruby.so.3.1.0: FAIL: notes test because gaps were detected in the annobin coverage Hardened: libruby.so.3.1.0: FAIL: cf-protection test because no .note.gnu.property section = no control flow information Hardened: libruby.so.3.1.0: FAIL: property-note test because no .note.gnu.property section found Hardened: Rerun annocheck with --verbose to see more information on the tests. ``` The message `Hardened: libruby.so.3.1.0: FAIL: property-note test because no .note.gnu.property section found` is what I found in our internal test. For other FAIL messages, maybe it can be fixed by changing how to build. Asking a colleague, I was told that the `coroutine/*/Context.S` files such as [coroutine/x86/Context.S](https://github.com/ruby/ruby/blob/master/coroutine… cause the failure. Do you have any idea how to fix this? Thanks. * [1] https://sourceware.org/annobin/ * [2] You can see `man annocheck` or https://www.mankier.com/1/annocheck . ---Files-------------------------------- 0001-Add-.note.gnu.property-sections.patch (2.64 KB) 0001-Add-.note.gnu.property-sections.patch (3.69 KB) config-pie.log (11.4 KB) -- https://bugs.ruby-lang.org/

1 0

[ruby-core:118259] [Ruby master Feature#20567] Net::HTTPResponse#detect_encoding should default to UTF-8 for application/javascript
by postmodern (Hal Brodigan) 09 Jun '24

09 Jun '24

Issue #20567 has been reported by postmodern (Hal Brodigan). ---------------------------------------- Feature #20567: Net::HTTPResponse#detect_encoding should default to UTF-8 for application/javascript https://bugs.ruby-lang.org/issues/20567 * Author: postmodern (Hal Brodigan) * Status: Open ---------------------------------------- It is fairly common now for JavaScript to include UTF-8 characters. If a misconfigured web server returns an `application/javascript` response for a JavaScript file, but *without* `charset=utf-8` in the `Content-Type`, `Net::HTTPResponse` will return a response body with encoding of ASCII-8BIT. This can cause lots of subtle String parsing/encoding issues when the response body is parsed or transformed. `Net::HTTPResponse#detect_encoding` should default the response encoding to UTF-8 if the `Content-Type` includes `application/javascript`. [RFC 4329](https://www.rfc-editor.org/rfc/rfc4329#section-4) also seems to encourage defaulting to UTF-8. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:118257] [Ruby master Bug#20566] string << 0xC2 should raise a RangeError if the string encoding is Encoding::ASCII
by postmodern (Hal Brodigan) 09 Jun '24

09 Jun '24

Issue #20566 has been reported by postmodern (Hal Brodigan). ---------------------------------------- Bug #20566: string << 0xC2 should raise a RangeError if the string encoding is Encoding::ASCII https://bugs.ruby-lang.org/issues/20566 * Author: postmodern (Hal Brodigan) * Status: Open * ruby -v: ruby 3.4.0preview1 (2024-05-16 master 9d69619623) [x86_64-linux] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- I noticed that appending extended ASCII characters (0x80 - 0xff) works when the String's encoding is Encoding::ASCII, as well as Encoding::ASCII_8BIT. I was under the impression that Encoding::ASCII only covers the regular ASCII character set (0x00 - 0x7F). I believe appending 0xC2 to a String with Encoding::ASCII should raise a RangeError. ## Steps To Reproduce ```ruby str = String.new(encoding: Encoding::ASCII) str << 0xC2 ``` ### Expected Result ``` RangeError: character 194 is out of range ``` ### Actual Result ```ruby "\xC2" ``` -- https://bugs.ruby-lang.org/

2 1