- ruby-core - ml.ruby-lang.org

[ruby-core:118315] [Ruby master Bug#20579] `Ripper.tokenize("1\n ")` does not include trailing spaces
by tompng (tomoya ishida) 13 Jun '24

13 Jun '24

Issue #20579 has been reported by tompng (tomoya ishida). ---------------------------------------- Bug #20579: `Ripper.tokenize("1\n ")` does not include trailing spaces https://bugs.ruby-lang.org/issues/20579 * Author: tompng (tomoya ishida) * Status: Open * ruby -v: ruby 3.4.0dev (2024-06-13T09:49:46Z master 8b843b0dc7) [x86_64-linux] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- ~~~ruby # trailing :on_sp, :on_ignored_nl are included in the result Ripper.tokenize("1\n\t \t\n\t \t") # => ["1", "\n", "\t \t", "\n", "\t \t"] # trailing :on_sp disappears Ripper.tokenize("1\n\t \t") # => ["1", "\n"] ~~~ Found with `Ripper.tokenize(random_string).join != random_string` while investigating https://bugs.ruby-lang.org/issues/20578 -- https://bugs.ruby-lang.org/

1 0

[ruby-core:118314] [Ruby master Bug#20578] Tokenizing string literal that have newline and invalid escape is wrong
by tompng (tomoya ishida) 13 Jun '24

13 Jun '24

Issue #20578 has been reported by tompng (tomoya ishida). ---------------------------------------- Bug #20578: Tokenizing string literal that have newline and invalid escape is wrong https://bugs.ruby-lang.org/issues/20578 * Author: tompng (tomoya ishida) * Status: Open * ruby -v: ruby 3.4.0dev (2024-06-13T09:49:46Z master 8b843b0dc7) [x86_64-linux] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- Tokenizing string literal that have newline and invalid escape is wrong When a string literal includes `\n` and an invalid escape after it, tokenize result gets wrong. ~~~ruby Ripper.tokenize "\"hello\\x world" # => ["\"", "hello\\x", " world"] # looks good Ripper.tokenize "\"\nhello\\x world" # => ["\"", "\n world", "hello\\x"] # order is reversed ~~~ These invalid escapes also gets wrong ~~~ruby Ripper.tokenize("\"\n\\Cxx\"") #=> ["\"", "\nx", "\\Cx", "\""] Ripper.tokenize("\"\n\\Mxx\"") #=> ["\"", "\nx", "\\Mx", "\""] Ripper.tokenize("\"\n\\c\\cx\"") #=> ["\"", "\nx", "\\c\\c", "\""] Ripper.tokenize("\"\n\\ux\"") #=> ["\"", "\nx", "\""] Ripper.tokenize("\"\n\\xx\"") #=> ["\"", "\nx", "\\x", "\""] ~~~ And these literals also gets wrong ~~~ruby Ripper.tokenize("<<A\n\n\\xyz") #=> ["<<A", "\n", "\nyz", "\\x"] Ripper.tokenize("%(\n\\xyz)") #=> ["%(", "\nyz", "\\x", ")"] Ripper.tokenize("%Q(\n\\xyz)") #=> ["%Q(", "\nyz", "\\x", ")"] Ripper.tokenize(":\"\n\\xyz\"") #=> [":\"", "\nyz", "\\x", "\""] ~~~ I encountered this while typing a valid string literal into IRB ~~~ruby irb(main):001> " irb(main):002> \x█ ~~~ Other invalid escape sequence that disappears from tokenize result ~~~ruby Ripper.tokenize('"\u{123') # => ["\""] ~~~ -- https://bugs.ruby-lang.org/

1 0

[ruby-core:118267] [Ruby master Misc#20568] Look for test.rb in the current working directory
by eightbitraptor (Matthew Valentine-House) 12 Jun '24

12 Jun '24

Issue #20568 has been reported by eightbitraptor (Matthew Valentine-House). ---------------------------------------- Misc #20568: Look for test.rb in the current working directory https://bugs.ruby-lang.org/issues/20568 * Author: eightbitraptor (Matthew Valentine-House) * Status: Open ---------------------------------------- [Github PR link](https://github.com/ruby/ruby/pull/10950) The `test.rb` script used by `make run` etc is explicitly located in the `$srcdir` - the location of the Ruby source code. The [Building Ruby](https://docs.ruby-lang.org/en/master/contributing/building_ruby_md.ht… documentation recommends using out-of-source builds. Requiring `test.rb` to be located in the source directory prevents me from having multiple build directories, with a distinct `test.rb` per build, instead forcing pollution of the source directory. I propose looking for `test.rb` in the current working directory. This will maintain compatibility with in-source builds, and provide extra flexibility for those of us using out-of-source builds. There will be a small one time impact for Ruby committers, who will have to move their existing `test.rb` files into their build directories. -- https://bugs.ruby-lang.org/

2 1

[ruby-core:118302] [Ruby master Bug#20577] FREE_AT_EXIT segfaulting after 32683aa1
by mdalessio (Mike Dalessio) 12 Jun '24

12 Jun '24

Issue #20577 has been reported by mdalessio (Mike Dalessio). ---------------------------------------- Bug #20577: FREE_AT_EXIT segfaulting after 32683aa1 https://bugs.ruby-lang.org/issues/20577 * Author: mdalessio (Mike Dalessio) * Status: Open * ruby -v: ruby 3.4.0dev (2024-06-11T14:33:51Z master 32683aa18d) [x86_64-linux] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- After commit 32683aa1 on master, using `RUBY_FREE_AT_EXIT` results in a segfault. When run with valgrind, many invalid reads are reported. Repro with ``` RUBY_FREE_AT_EXIT=1 bundle exec /path/to/ruby -e 'puts RUBY_DESCRIPTION' ``` or for valgrind details ``` RUBY_FREE_AT_EXIT=1 bundle exec valgrind /path/to/ruby -e 'puts RUBY_DESCRIPTION' ``` Typical example valgrind error is: ``` ==441955== Invalid read of size 4 ==441955== at 0x2C6F97: hash_table_index (id_table.c:132) ==441955== by 0x2C6F97: rb_id_table_lookup (id_table.c:230) ==441955== by 0x31AB50: cached_callable_method_entry (vm_method.c:1415) ==441955== by 0x31AB50: callable_method_entry_or_negative (vm_method.c:1485) ==441955== by 0x31C4EE: callable_method_entry (vm_method.c:1522) ==441955== by 0x31C4EE: rb_callable_method_entry (vm_method.c:1529) ==441955== by 0x18F370: obj_any_hash (hash.c:222) ==441955== by 0x19F444: any_hash (hash.c:204) ==441955== by 0x19F444: rb_any_hash (hash.c:260) ==441955== by 0x19F444: ar_do_hash (hash.c:418) ==441955== by 0x19F444: ar_delete (hash.c:1093) ==441955== by 0x19F444: rb_hash_stlike_delete (hash.c:2385) ==441955== by 0x1A0CCB: rb_hash_delete_entry (hash.c:2402) ==441955== by 0x2C59B6: rb_gc_free_dsymbol (symbol.c:851) ==441955== by 0x17AC53: obj_free (gc.c:3460) ==441955== by 0x17D045: rb_objspace_call_finalizer_i (gc.c:4352) ==441955== by 0x17D045: gc_each_object (gc.c:4302) ==441955== by 0x17D045: rb_objspace_call_finalizer (gc.c:4401) ==441955== by 0x162719: rb_ec_finalize (eval.c:164) ==441955== by 0x162719: rb_ec_cleanup (eval.c:255) ==441955== by 0x15CE09: rb_main (main.c:40) ==441955== by 0x15CE09: main (main.c:59) ==441955== Address 0x4ec4610 is 0 bytes inside a block of size 24 free'd ==441955== at 0x484988F: free (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so) ==441955== by 0x175BD1: objspace_xfree (gc.c:12153) ==441955== by 0x175BD1: objspace_xfree (gc.c:12084) ==441955== by 0x175BD1: ruby_sized_xfree.constprop.0 (gc.c:12257) ==441955== by 0x17A9A0: obj_free (gc.c:3293) ==441955== by 0x17D045: rb_objspace_call_finalizer_i (gc.c:4352) ==441955== by 0x17D045: gc_each_object (gc.c:4302) ==441955== by 0x17D045: rb_objspace_call_finalizer (gc.c:4401) ==441955== by 0x162719: rb_ec_finalize (eval.c:164) ==441955== by 0x162719: rb_ec_cleanup (eval.c:255) ==441955== by 0x15CE09: rb_main (main.c:40) ==441955== by 0x15CE09: main (main.c:59) ==441955== Block was alloc'd at ==441955== at 0x4846828: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so) ==441955== by 0x189B0A: objspace_xmalloc0 (gc.c:11938) ==441955== by 0x2C6EA6: rb_id_table_create (id_table.c:98) ==441955== by 0x31AD99: cache_callable_method_entry (vm_method.c:1444) ==441955== by 0x31AD99: callable_method_entry_or_negative (vm_method.c:1502) ==441955== by 0x32A7DE: callable_method_entry (vm_method.c:1522) ==441955== by 0x32A7DE: vm_respond_to (vm_method.c:2878) ==441955== by 0x3305E3: check_funcall_respond_to (vm_eval.c:602) ==441955== by 0x3305E3: rb_check_funcall_default_kw (vm_eval.c:676) ==441955== by 0x20E896: convert_type_with_id (object.c:3131) ==441955== by 0x20E896: rb_check_convert_type_with_id (object.c:3223) ==441955== by 0x1672A7: rb_stat (file.c:1299) ==441955== by 0x16E15E: rb_file_exist_p (file.c:1805) ==441955== by 0x16E15E: rb_check_realpath_internal (file.c:4567) ==441955== by 0x1D6639: get_loaded_features_index (load.c:415) ==441955== by 0x1D6FB7: rb_provide_feature (load.c:703) ==441955== by 0x1A2629: rb_call_inits (inits.c:64) ==441955== by 0x162B67: ruby_setup (eval.c:86) ==441955== by 0x162C1C: ruby_init (eval.c:98) ==441955== by 0x15CDF7: rb_main (main.c:39) ==441955== by 0x15CDF7: main (main.c:59) ``` I'm attaching a full log from my system. cc @peterzhu2118 ---Files-------------------------------- bug.log (238 KB) -- https://bugs.ruby-lang.org/

2 5

[ruby-core:116790] [Ruby master Bug#20270] Options with `--parser=prism`
by nobu (Nobuyoshi Nakada) 11 Jun '24

11 Jun '24

Issue #20270 has been reported by nobu (Nobuyoshi Nakada). ---------------------------------------- Bug #20270: Options with `--parser=prism` https://bugs.ruby-lang.org/issues/20270 * Author: nobu (Nobuyoshi Nakada) * Status: Open * Priority: Normal * Backport: 3.0: DONTNEED, 3.1: DONTNEED, 3.2: DONTNEED, 3.3: REQUIRED ---------------------------------------- ### `--dump` option Currently `parsetree` and `prism_parsetree` bits are separated, but it seems meaningless as far as `--parser` option selects only one parser. Why doesn't simply `--dump=parsetree` dump AST by parse.y, or PRISM AST if `--parser=prism` is given? Actually, `ruby --dump=parsetree --parser=prism` just segfaults, because these bits and `--parser` option are separately handled. ### streaming code from stdin ```sh-session $ echo end | ruby --parser=prism - ruby: warning: The compiler based on the Prism parser is currently experimental and compatibility with the compiler based on parse.y is not yet complete. Please report any issues you find on the `ruby/prism` issue tracker. -: warning: Prism support for streaming code from stdin is not currently supported ``` Nothing reported is fine, since it is not currently supported. I was a bit curious that it looks like trying something after the warning. ```C if (strcmp(opt->script, "-") == 0) { int xflag = opt->xflag; VALUE rb_source = open_load_file(opt->script_name, &xflag); opt->xflag = xflag != 0; rb_warn("Prism support for streaming code from stdin is not currently supported"); error = pm_parse_string(&result, rb_source, opt->script_name); } ``` Note that `open_load_file` returns `rb_stdin` when `-` is given. This object is a `RFile`, not a `RString`, and `RSTRING_PTR(rb_source)` accesses after the object boundary. It should just bail out, not try obviously wrong thing. ### Fix https://github.com/nobu/ruby/tree/options-refactor -- https://bugs.ruby-lang.org/

2 3

[ruby-core:118154] [Ruby master Bug#20521] Memory leak in Ripper parsing
by kddnewton (Kevin Newton) 11 Jun '24

11 Jun '24

Issue #20521 has been reported by kddnewton (Kevin Newton). ---------------------------------------- Bug #20521: Memory leak in Ripper parsing https://bugs.ruby-lang.org/issues/20521 * Author: kddnewton (Kevin Newton) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- Currently, this code is leaking: ```ruby require "ripper" source = <<~'RUBY' <<-A; %w[j\ i A j] RUBY 10.times do 100_000.times do Ripper.sexp_raw(source) end puts `ps -o rss= -p #{$$}` end ``` This results in: ``` 31584 36304 41504 47008 51712 56384 61088 65792 70480 75200 ``` I haven't managed to track it down, but I can see that 3.2.2 does not leak but 3.3.0 does leak. -- https://bugs.ruby-lang.org/

3 2

[ruby-core:118173] [Ruby master Misc#20524] mswin and gmp
by MSP-Greg (Greg L) 11 Jun '24

11 Jun '24

Issue #20524 has been reported by MSP-Greg (Greg L). ---------------------------------------- Misc #20524: mswin and gmp https://bugs.ruby-lang.org/issues/20524 * Author: MSP-Greg (Greg L) * Status: Open ---------------------------------------- Currently, the `vcpkg.json` file exists, and it can be used to specify what MSFT/vcpkg packages to install, and also a repo commit, which will determine the what package versions are used. The file lists the gmp package. 1. To use the gmp package in mswin, `win32/configure.bat` shows an option of `--with-gmp`. This option is not used in the `windows.yml` workflow file, nor is it mentioned in `doc/windows.md`. 2. When using the option, the build fails when compiling `bignum.c`, as it can’t find `gmp.h`. It appears the vcpkg include folder is not being passed to `cl`? So, it seems a few things are amiss. `gmp` was added a long time ago. With current mswin builds, will `gmp` improve `Bignum`’s performance? Is that unknown? If there are benchmarks to determine this, I could certainly run them… -- https://bugs.ruby-lang.org/

3 2

[ruby-core:118273] [Ruby master Bug#20572] Compilation hangs with nested until
by kddnewton (Kevin Newton) 11 Jun '24

11 Jun '24

Issue #20572 has been reported by kddnewton (Kevin Newton). ---------------------------------------- Bug #20572: Compilation hangs with nested until https://bugs.ruby-lang.org/issues/20572 * Author: kddnewton (Kevin Newton) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- ```ruby until while 1 end end ``` this code causes an infinite loop in the compiler. -- https://bugs.ruby-lang.org/

2 2

[ruby-core:118275] [Ruby master Feature#19001] Data: Add #to_h symmetric to constructor with keyword args [Follow-on to #16122 Data: simple immutable value object]
by RubyBugs (A Nonymous) 10 Jun '24

10 Jun '24

Issue #19001 has been updated by RubyBugs (A Nonymous). Agreed, this need ended up being met by the initial implementation! Thank you, closing this ticket. zverok (Victor Shepelev) wrote in #note-2: > There isn't any need for this ticket as a separate request, as far as I am concerned. > It works in the initial implementation of data already as submitted in https://bugs.ruby-lang.org/issues/16122#note-68, and even works with ol' good Struct, too: > ```ruby > Point = Struct.new(:x, :y, :z) > > Point[1, 0, 0].to_h.then { Point[**_1] } == Point[1, 0, 0] # => true > ``` ---------------------------------------- Feature #19001: Data: Add #to_h symmetric to constructor with keyword args [Follow-on to #16122 Data: simple immutable value object] https://bugs.ruby-lang.org/issues/19001#change-108768 * Author: RubyBugs (A Nonymous) * Status: Open ---------------------------------------- *Extracted a follow-up to [#16122 Data: simple immutable value object](https://bugs.ruby-lang.org/issues/16122)* # Proposal: Add a `#to_h` method symmetric to a constructor accepting keyword arguments This allows round-trip between a `Hash` and a Value object instance, for example: ```ruby Point = Data.define(:x, :y, :z) points = [ Point.new(x: 1, y: 0, z: 0), Point.new(x: 0, y: 1, z: 0), Point.new(x: 0, y: 0, z: 1), ] hashes = points.map(&:to_h) points_2 = hashes.map { |h| Point.new(**h) } points_2 == points #=> true ``` ## Why? Having symmetric operation between `#to_h` and a keyword-args constructor is a major ergonomic factor in usage of immutable value objects. To play with code that works like this, you may take a look at the [Values gem](https://rubygems.org/gems/values) ## Alternatives If there is no symmetric construction and de-construction along these lines, a number of use cases become more complicated and less ergonomic. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:116039] [Ruby master Bug#20154] aarch64: configure overrides `-mbranch-protection` if it was set in CFLAGS via environment
by jprokop (Jarek Prokop) 09 Jun '24

09 Jun '24

Issue #20154 has been reported by jprokop (Jarek Prokop). ---------------------------------------- Bug #20154: aarch64: configure overrides `-mbranch-protection` if it was set in CFLAGS via environment https://bugs.ruby-lang.org/issues/20154 * Author: jprokop (Jarek Prokop) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- Recently a GH PR was merged <https://github.com/ruby/ruby/pull/9306> For PAC/BTI support on ARM CPUs for Coroutine.S. Without proper compilation support in configure.ac it segfaults Ruby with fibers on CPUs where PAC is supported: https://bugs.ruby-lang.org/issues/20085 At the time of writing, configure.ac appends the first option from a list for flag `-mbranch-protection` that successfully compiles a program <https://github.com/ruby/ruby/blob/master/configure.ac#L829>, to XCFLAGS and now also ASFLAGS to fix issue 20085 for Ruby master. This is suboptimal for Fedora as we set -mbranch-protection=standard by default in C{,XX}FLAGS: ``` CFLAGS='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Werror=implicit-function-declaration -Werror=implicit-int -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -mbranch-protection=standard -fasynchronous-unwind-tables -fstack-clash-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer ' export CFLAGS CXXFLAGS='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -mbranch-protection=standard -fasynchronous-unwind-tables -fstack-clash-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer' export CXXFLAGS ``` And the appended flag overrides distribution's compilation configuration, which in this case ends up omitting BTI instructions and only using PAC. Would it make sense to check if such flags exist and not overwrite them if they do? Serious proposals: 1. Simplest fix that does not overwrite what is set in the distribution and results in higher security is simply prepending the list of options with `-mbranch-protection=standard`, it should cause no problems on ARMv8 CPUs and forward, BTI similarly to PAC instructions result into NOP, it is only extending the capability. See attached 0001-aarch64-Check-mbranch-protection-standard-first-to-u.patch 2. Other fix that sounds more sane IMO and dodges this kind of guessing where are all the correct places for the flag is what another Fedora contributor Florian Weimer suggested: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org… "The reliable way to do this would be to compile a C file and check whether that enables __ARM_FEATURE_PAC_DEFAULT, and if that's the case, define a *different* macro for use in the assembler implementation. This way, you don't need to care about the exact name of the option." IOW instead of using __ARM_FEATURE_* directly in that code, define a macro in the style of "USE_PAC" with value of the feature if it is defined, I think that way we shouldn't need to append ASFLAGS anymore. However it's also important to catch the value of those macros as their values have meaning, I have an idea how to do that but I'd get on that monday earliest. ---Files-------------------------------- 0001-aarch64-Check-mbranch-protection-standard-first-to-u.patch (1004 Bytes) -- https://bugs.ruby-lang.org/

4 10