February 2023 - ruby-core - ml.ruby-lang.org

[ruby-core:112200] [Ruby master Feature#19406] Allow declarative reference definition for rb_typed_data_struct
by eightbitraptor (Matthew Valentine-House) 12 Mar '23

12 Mar '23

Issue #19406 has been reported by eightbitraptor (Matthew Valentine-House). ---------------------------------------- Feature #19406: Allow declarative reference definition for rb_typed_data_struct https://bugs.ruby-lang.org/issues/19406 * Author: eightbitraptor (Matthew Valentine-House) * Status: Open * Priority: Normal ---------------------------------------- [Github PR 7153](https://github.com/ruby/ruby/pull/7153) ## Summary This PR proposes an additional API for C extension authors to define wrapped struct members that point to Ruby objects, when the struct being wrapped contains only members with primitive types (ie. no arrays or unions). The new interface passes an offset from the top of the data structure, rather than the reference `VALUE` itself, allowing the GC to manipulate both the reference edge (the address holding the pointer), as well as the underlying object. This allows Ruby's GC to handle marking, object movement and reference updating independently without calling back into user supplied code. ## Implementation When a wrapped struct contains a simple list of members (such as the `struct enumerator` in `enumerator.c`). We can declare all of the struct members that may point to valid Ruby objects as `RUBY_REF_EDGE` in a static array. If we choose to do this, then we can mark the corresponding `rb_data_type_t` as `RUBY_TYPED_DECL_MARKING` and pass a pointer to the references array in the `data` field. To avoid having to also find space in the `rb_data_type_t` to define a length for the references list, I've chosen to require list termination with `RUBY_REF_END` - defined as `UINTPTR_MAX`. My assumption is that no single wrapped struct will ever be large enough that `UINTPTR_MAX` is actually a valid reference. We don't have to then define `dmark` or `dcompact` callback functions. Marking, object movement, and reference updating will be handled for us by the GC. ```C struct enumerator { VALUE obj; ID meth; VALUE args; VALUE fib; VALUE dst; VALUE lookahead; VALUE feedvalue; VALUE stop_exc; VALUE size; VALUE procs; rb_enumerator_size_func *size_fn; int kw_splat; }; static const size_t enumerator_refs[] = { RUBY_REF_EDGE(enumerator, obj), RUBY_REF_EDGE(enumerator, args), RUBY_REF_EDGE(enumerator, fib), RUBY_REF_EDGE(enumerator, dst), RUBY_REF_EDGE(enumerator, lookahead), RUBY_REF_EDGE(enumerator, feedvalue), RUBY_REF_EDGE(enumerator, stop_exc), RUBY_REF_EDGE(enumerator, size), RUBY_REF_EDGE(enumerator, procs), RUBY_REF_END }; static const rb_data_type_t enumerator_data_type = { "enumerator", { NULL, enumerator_free, enumerator_memsize, NULL, }, 0, (void *)enumerator_refs, RUBY_TYPED_FREE_IMMEDIATELY | RUBY_TYPED_DECL_MARKING }; ``` ### Benchmarking Benchmarking shows that this reference declaration style does not degrade performance when compared to the callback style. To benchmark this we created a C extension that initialized a struct with 20 `VALUE` members, all set to point to Ruby strings. We wrapped each struct using `TypedData_Make_Struct` in an object. One object was configured with callback functions and one was configured with declarative references. In separate scripts we then created 500,000 of these objects, added them to a list, so they would be marked and not swept and used `GC.verify_compaction_references` to make sure everything that could move, did. Finally we created a wrapper script that used seperate processes to run each GC type (to ensure that the GC's were completely independent), ran each benchmark 50 times, and collected the results of `GC.stat[:time]`. We did this on an M1 Pro MacBook (aarch64), and a Ryzen 3600 We then plotted the results: ![chart showing GC time between callback and declarative marking on arm64 and x86_64](https://user-images.githubusercontent.com/31869/216573409-ddafa3bd-… As we can see from this, there has been no real impact to GC performance in our benchmarks. Benchmark code and harnesses is [available in this Github repo](https://github.com/eightbitraptor/test_decl_marking) ## Justification Requiring extension authors to implement seperate `dmark` and `dcompact` callbacks can be error-prone, and pushes GC responsibilities from the GC into user supplied code. This can be a source of bugs arising from the `dmark` and `dcompact` functions being implemented incorrectly, or becoming out of sync with each other. There has already been work done by @Peterzhu2118 [to try and unify these callbacks](https://github.com/ruby/ruby/pull/7140), so that authors can define a single function, that will be used for both marking and compacting, removing the risk of these callbacks becoming out of sync. This proposal works alongside Peter's earlier work to eliminate the callbacks entirely for the "simple reference" case. This means that extension authors with simple structs to wrap can declare which of their struct members point to Ruby objects to get GC marking and compaction support. And extension authors with more complex requirements will only have to implement a single function, using Peter's work. In addition to this, passing the GC the address of a reference rather than the reference itself (edge based, rather than object based enqueing), allows the GC itself to have more control over how it manipulates that reference. This means that when considering alternative GC implementations for Ruby (such as our [ongoing work integrating MMTk into Ruby](https://github.com/mmtk/mmtk-ruby)[^1] We don't need to call from Ruby into library code, and then back into Ruby code as often; which can increase performance, and allow more complex algorithms to be implemented. [^1]: [MMtk](https://www.mmtk.io/) is the Memory Management Toolkit. A framework for implementing automatic memory management strategies ## Trade-offs This PR provides another method for defining references in C extensions, in addition to the callback based approach, effectively widening the extension API. Extension authors will now need to choose whether to use the declarative approach, or a callback based approach depending on their use case. This is more complex for extension authors. However because the callback's do still exist, this does mean that extension authors can migrate their own code to this new, faster approach at their leisure. ## Further work As part of this work we inspected all uses of `rb_data_type_t` in the Ruby source code and of 134 separete instances, 60 wrapped structs that contained `VALUE` members that could point to Ruby objects. Out of these 27 were "simple" structs that would benefit from this approach, 28 contained complex references (unions, loops etc) that won't work with this approach, and 5 were situations that were unsure, that we believe we could make work given some slight refactors. -- https://bugs.ruby-lang.org/

5 5

[ruby-core:112427] [Ruby master Feature#19437] Add marking and sweeping time to GC.stat
by peterzhu2118 (Peter Zhu) 10 Mar '23

10 Mar '23

Issue #19437 has been reported by peterzhu2118 (Peter Zhu). ---------------------------------------- Feature #19437: Add marking and sweeping time to GC.stat https://bugs.ruby-lang.org/issues/19437 * Author: peterzhu2118 (Peter Zhu) * Status: Open * Priority: Normal ---------------------------------------- Pull Request: https://github.com/ruby/ruby/pull/7304 There is a `time` key in GC.stat that gives us the total time spent in GC. However, we don't know what proportion of the time is spent between marking and sweeping. This makes it difficult to tune the GC as we're not sure where to focus our efforts on. This PR adds keys `marking_time` and `sweeping_time` to GC.stat for the time spent marking and sweeping, in milliseconds. -- https://bugs.ruby-lang.org/

6 6

[ruby-core:112536] [Ruby master Feature#19457] Some improvements to test-all suite
by luke-gru (Luke Gruber) 09 Mar '23

09 Mar '23

Issue #19457 has been reported by luke-gru (Luke Gruber). ---------------------------------------- Feature #19457: Some improvements to test-all suite https://bugs.ruby-lang.org/issues/19457 * Author: luke-gru (Luke Gruber) * Status: Open * Priority: Normal ---------------------------------------- Hi, I've been making some improvements to this suite in a branch of my own and came up with a list of changes I think are good but I would like input from developers. 1) Currently test-all with -j option does not show errors/failures immediately, it waits until end. Additionaly it doesn't really show them at end, it only shows after it retries the tests in current process. I made changes to make it show immediately and have a summary before the retry. 2) If timeout occurs in -j mode (default 180 sec), the worker process is terminated but no new worker is added to replace it. I added this. 3) If a timeout occurs in -j mode, it is not reported immediately. I made it do so. 4) I added -no-leakchecker flag. I use this often when I don't want any output of DRB tests because I get lots of leakchecker reports from this for some reason. 5) when filtering test name with --name option, you can give regexp like /test_hash/ but it can also match class name like !/DRbTest/. You can't give //i flag so you need to get case exactly right for the class names. I added //i flag functionality for positive and negative regexps. 6) Fixed colorization of failures/skips. 7) I added test suite (test class) GC in serial (no -j) mode. After a test class finishes the constant is removed and references deleted. It saves quite a bit of memory in my tests so far. Motivation to reduce memory and reduce full GC pause time. I have all these changes in a single branch. If any of these changes interest you, let me know and I will try to isolate them in a single commit and make a PR. -- https://bugs.ruby-lang.org/

3 2

[ruby-core:112284] [Ruby master Bug#19424] Degradation in **Marshal load** only in Ruby 3.1.2 compared to 2.7.4
by sumitdey035 (Sumit Dey) 09 Mar '23

09 Mar '23

Issue #19424 has been reported by sumitdey035 (Sumit Dey). ---------------------------------------- Bug #19424: Degradation in **Marshal load** only in Ruby 3.1.2 compared to 2.7.4 https://bugs.ruby-lang.org/issues/19424 * Author: sumitdey035 (Sumit Dey) * Status: Open * Priority: Normal * ruby -v: 3.1.2 * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- I can see degradation in **Marshal load** only in Ruby 3.1.2 compared to 2.7.4 Processing time increased by 200%(2.4 sec to 4.3 sec) Memory allocation increased by 600%(6500001 to 39000004) ``` require 'benchmark' require 'ostruct' N_OJ = 500_000 ex_HASH = { 'one' => 1, 'array' => [ true, false ] } ex_JSON = '{ "one": 1, "array": [ true, false ] }' ex_STRUCT = OpenStruct.new( one: 1, hash: ex_HASH, array: [ true, false ] ) ex_MARSHAL = "\x04\bU:\x0FOpenStruct{\b:\bonei\x06:\thash{\aI\"\bone\x06:\x06ETi\x06I\"\narray\x06;\bT[\aTF:\narray[\aTF" "-----------------Ruby #{system("rbenv version")}----------------" Benchmark.bm(20) do |x| x.report('native marshal dump') do N_OJ.times do y = Marshal.dump(ex_STRUCT) end end x.report('native marshal load') do N_OJ.times do y = Marshal.load(ex_MARSHAL) end end start_memory = GC.stat[:total_allocated_objects] N_OJ.times do y = Marshal.dump(ex_STRUCT) end end_memory = GC.stat[:total_allocated_objects] print "Marshal dump memory allocation- #{end_memory - start_memory}\n" start_memory = GC.stat[:total_allocated_objects] N_OJ.times do y = Marshal.load(ex_MARSHAL) end end_memory = GC.stat[:total_allocated_objects] print "Marshal load memory allocation- #{end_memory - start_memory}\n" end``` **Benchmark and Memory consumption result** ![](Screenshot 2023-02-07 at 1.04.49 PM.png) ---Files-------------------------------- Screenshot 2023-02-07 at 1.04.49 PM.png (184 KB) -- https://bugs.ruby-lang.org/

5 13

[ruby-core:112494] [Ruby master Feature#19450] Is there an official way to set a class name without setting a constant?
by ioquatix (Samuel Williams) 09 Mar '23

09 Mar '23

Issue #19450 has been reported by ioquatix (Samuel Williams). ---------------------------------------- Feature #19450: Is there an official way to set a class name without setting a constant? https://bugs.ruby-lang.org/issues/19450 * Author: ioquatix (Samuel Williams) * Status: Open * Priority: Normal ---------------------------------------- This is the best I could come up with: ```ruby klass = Class.new Object.const_set("Klass", klass) Object.send(:remove_const, "Klass") puts klass.new # => #<Klass:0x0000000100a9d688> ``` Can we do better? -- https://bugs.ruby-lang.org/

9 26

[ruby-core:112333] [Ruby master Feature#19432] Introduce a wrapping operator (&) to Proc
by joel＠drapper.me (Joel Drapper) 08 Mar '23

08 Mar '23

Issue #19432 has been reported by joel(a)drapper.me (Joel Drapper). ---------------------------------------- Feature #19432: Introduce a wrapping operator (&) to Proc https://bugs.ruby-lang.org/issues/19432 * Author: joel(a)drapper.me (Joel Drapper) * Status: Open * Priority: Normal ---------------------------------------- I don't know if this concept exists under another name, or whether there’s a technical term for it. I often find myself wanting to wrap a proc in another proc. Here's a snippet from a recent example where a visitor class renders a TipTap AST. Given a `text` node, we want to output the text by calling the `text` method with the value. But if the text has `marks`, we want to iterate through each mark, wrapping the output for each mark. It's possible to do this using the `>>=` operator if each proc explicitly returns another proc. ```ruby when "text" result = -> { -> { text node["text"] } } node["marks"]&.each do |mark| case mark["type"] when "bold" result >>= -> (r) { -> { strong { r.call } } } when "italic" result >>= -> (r) { -> { em { r.call } } } end end result.call.call end ``` This is quite difficult to follow and the `result.call.call` feels wrong. I think the concept of wrapping one proc in another proc would make for a great addition to `Proc` itself. I prototyped this using the `&` operator. ```ruby class Proc def &(other) -> { other.call(self) } end end ``` With this definition, we can call `&` on the original proc with our other proc to return a new proc that calls the other proc with the original proc as an argument. It also works with `&=`, so the above code can be refactored to this: ```ruby when "text" result = -> { text node["text"] } node["marks"]&.each do |mark| case mark["type"] when "bold" result &= -> (r) { strong { r.call } } when "italic" result &= -> (r) { em { r.call } } end end result.call end ``` -- https://bugs.ruby-lang.org/

4 4

[ruby-core:112584] [Ruby master Feature#19465] [PATCH] reuse open(2) from rb_file_load_ok on POSIX-like system
by normalperson (Eric Wong) 08 Mar '23

08 Mar '23

Issue #19465 has been reported by normalperson (Eric Wong). ---------------------------------------- Feature #19465: [PATCH] reuse open(2) from rb_file_load_ok on POSIX-like system https://bugs.ruby-lang.org/issues/19465 * Author: normalperson (Eric Wong) * Status: Open * Priority: Normal ---------------------------------------- ``` When loading Ruby source files, we can save the result of successful opens as open(2)/openat(2) are a fairly expensive syscalls. This also avoids a time-of-check-to-time-of-use (TOCTTOU) problem. This reduces open(2) syscalls during `require'; but should be most apparent when users have a small $LOAD_PATH. Users with large $LOAD_PATH will benefit less since there'll be more open(2) failures due to ENOENT. With `strace -c -e openat ruby -e exit' under Linux, this results in a ~14% reduction of openat(2) syscalls (glibc uses openat(2) to implement open(2)). % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 0.00 0.000000 0 296 110 openat 0.00 0.000000 0 254 110 openat Additionally, the introduction of `struct ruby_file_load_state' may make future optimizations more apparent. This change cannot benefit binary (.so) loading since the dlopen(3) API requires a filename and I'm not aware of an alternative that takes a pre-existing FD. In typical situations, Ruby source files outnumber the mount of .so files. I've only tested this lightly on small apps since I don't have large codebases to test on. However, I think organizing various on-stack variables into `struct ruby_file_load_state' can be beneficial if we end up using io-uring on Linux. ``` ---Files-------------------------------- 0001-reuse-open-2-from-rb_file_load_ok-on-POSIX-like-syst.patch (15.2 KB) -- https://bugs.ruby-lang.org/

7 8

[ruby-core:111812] [Ruby master Bug#19340] Ruby master 'make install' not installing rbs gem
by MSP-Greg (Greg L) 08 Mar '23

08 Mar '23

Issue #19340 has been reported by MSP-Greg (Greg L). ---------------------------------------- Bug #19340: Ruby master 'make install' not installing rbs gem https://bugs.ruby-lang.org/issues/19340 * Author: MSP-Greg (Greg L) * Status: Open * Priority: Normal * ruby -v: ruby 3.3.0dev (2023-01-14T11:30:19Z master 1ddeb7473d) * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Last three daily builds of ruby-dev-builder have not included the rbs gem. See https://github.com/ruby/ruby-dev-builder/actions/runs/3919887598/jobs/67011… ruby/ruby PR https://github.com/ruby/ruby/pull/7114 ruby/ruby-dev-builder PR https://github.com/ruby/ruby-dev-builder/pull/2 checks all bin files work Added a patch to ruby-loco for it, so its builds are ok. -- https://bugs.ruby-lang.org/

3 7

[ruby-core:112560] [Ruby master Bug#19462] MJIT not enabled with universal macOS x86_64 + arm64 build
by benhamilton (Ben Hamilton) 08 Mar '23

08 Mar '23

Issue #19462 has been reported by benhamilton (Ben Hamilton). ---------------------------------------- Bug #19462: MJIT not enabled with universal macOS x86_64 + arm64 build https://bugs.ruby-lang.org/issues/19462 * Author: benhamilton (Ben Hamilton) * Status: Open * Priority: Normal * ruby -v: ruby 3.3.0dev (2023-02-21T21:59:54Z master ae9e1aee59) [universal.x86_64-darwin22] * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- I'm trying to build a universal macOS x86_64 + arm64 ruby from upstream @ ae9e1aee59b0db1e61aa0473556165f9fd719cde on x86_64 macOS Ventura 13.2, Xcode 14.2, Apple clang version 14.0.0 (clang-1400.0.29.202). When I run `autoconf && ./configure --with-arch=x86_64,arm64`, MJIT support is `yes` and YJIT support is `no` (I assume YJIT is disabled because I'm building for arm64): ``` Configuration summary for ruby version 3.3.0 * Installation prefix: /usr/local * exec prefix: ${prefix} * arch: universal-darwin22 * site arch: ${arch} * RUBY_BASE_NAME: ruby * ruby lib prefix: ${libdir}/${RUBY_BASE_NAME} * site libraries path: ${rubylibprefix}/${sitearch} * vendor path: ${rubylibprefix}/vendor_ruby * target OS: darwin22 * compiler: clang * with thread: pthread * with coroutine: universal * enable shared libs: no * dynamic library ext: bundle * CFLAGS: -fdeclspec ${optflags} ${debugflags} ${warnflags} * LDFLAGS: -L. -fstack-protector-strong * DLDFLAGS: -Wl,-multiply_defined,suppress * optflags: -O3 -fno-fast-math * debugflags: -ggdb3 * warnflags: -Wall -Wextra -Wextra-tokens \ -Wdeprecated-declarations -Wdivision-by-zero \ -Wdiv-by-zero -Wimplicit-function-declaration \ -Wimplicit-int -Wmisleading-indentation \ -Wpointer-arith -Wshorten-64-to-32 \ -Wwrite-strings -Wold-style-definition \ -Wmissing-noreturn -Wno-cast-function-type \ -Wno-constant-logical-operand -Wno-long-long \ -Wno-missing-field-initializers \ -Wno-overlength-strings -Wno-parentheses-equality \ -Wno-self-assign -Wno-tautological-compare \ -Wno-unused-parameter -Wno-unused-value \ -Wunused-variable -Wundef * strip command: strip -A -n * install doc: rdoc * MJIT support: yes * YJIT support: no * man page type: doc * BASERUBY -v: ruby 2.6.10p210 (2022-04-12 revision 67958) \ [universal.x86_64-darwin22] ``` However, when I compile with `make -j2`, I see that it's disabled: ``` % make -j2 BASERUBY = /usr/bin/ruby --disable=gems CC = clang LD = ld LDSHARED = clang -dynamiclib CFLAGS = -fdeclspec -O3 -fno-fast-math -ggdb3 -Wall -Wextra -Wextra-tokens -Wdeprecated-declarations -Wdivision-by-zero -Wdiv-by-zero -Wimplicit-function-declaration -Wimplicit-int -Wmisleading-indentation -Wpointer-arith -Wshorten-64-to-32 -Wwrite-strings -Wold-style-definition -Wmissing-noreturn -Wno-cast-function-type -Wno-constant-logical-operand -Wno-long-long -Wno-missing-field-initializers -Wno-overlength-strings -Wno-parentheses-equality -Wno-self-assign -Wno-tautological-compare -Wno-unused-parameter -Wno-unused-value -Wunused-variable -Wundef -pipe -arch x86_64 -arch arm64 XCFLAGS = -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -fno-strict-overflow -fvisibility=hidden -DRUBY_EXPORT -fPIE -I. -I.ext/include/universal-darwin22 -I./include -I. -I./enc/unicode/15.0.0 CPPFLAGS = -D_XOPEN_SOURCE -D_DARWIN_C_SOURCE -D_DARWIN_UNLIMITED_SELECT -D_REENTRANT DLDFLAGS = -Wl,-multiply_defined,suppress -fstack-protector-strong -Wl,-pie -framework CoreFoundation -arch x86_64 -arch arm64 SOLIBS = -ldl -lobjc -lpthread LANG = en_US.UTF-8 LC_ALL = LC_CTYPE = MFLAGS = - --jobserver-fds=3,4 -j RUSTC = rustc YJIT_RUSTC_ARGS = --crate-name=yjit --crate-type=staticlib --edition=2021 -g -C lto=thin -C opt-level=3 -C overflow-checks=on '--out-dir=/Users/benhamilton/Developer/OpenSource/ruby/yjit/target/release/' ./yjit/src/lib.rs Apple clang version 14.0.0 (clang-1400.0.29.202) Target: x86_64-apple-darwin22.3.0 Thread model: posix InstalledDir: /Applications/Xcode_14.2.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin [snip] building .ext/include/universal-darwin22/rb_mjit_min_header-3.3.0-x86_64.h building .ext/include/universal-darwin22/rb_mjit_min_header-3.3.0-arm64.h This compiler (clang -fdeclspec -g -pipe -arch arm64 -w) looks not supported for MJIT. Giving up to generate MJIT header. This compiler (clang -fdeclspec -g -pipe -arch x86_64 -w) looks not supported for MJIT. Giving up to generate MJIT header. ``` The problem is here: * https://github.com/ruby/ruby/blob/1fdaa0666086529b3aae2d509a2e71c4247c3a12/… * https://github.com/ruby/ruby/blob/1fdaa0666086529b3aae2d509a2e71c4247c3a12/… * https://github.com/ruby/ruby/blob/1fdaa0666086529b3aae2d509a2e71c4247c3a12/… This logic runs `vm.c` through the C preprocessor and looks through the output for the `__clang__` preprocessor macro. However, `clang`'s C preprocessor only prints macro values with the `-dM` flag. We can add this flag, but it won't work with the MSVC compiler, so we need to only add it for gcc/clang. -- https://bugs.ruby-lang.org/

3 2

[ruby-core:112458] [Ruby master Bug#19444] YJIT String#+@ miscompilations
by alanwu (Alan Wu) 08 Mar '23

08 Mar '23

Issue #19444 has been reported by alanwu (Alan Wu). ---------------------------------------- Bug #19444: YJIT String#+@ miscompilations https://bugs.ruby-lang.org/issues/19444 * Author: alanwu (Alan Wu) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: REQUIRED ---------------------------------------- We noticed two issues with YJIT's `String#+@` (String unary plus) implementation. 1. It has issues with GC allocation tracing. We saw it SEGV with the `Stackprof` gem 2. It has a type confusion issue from assuming that the method always return an `::String` instance. It can return subclass instances The YJIT override does not exist on the 3.1 series. -- https://bugs.ruby-lang.org/

2 1