September 2024 - ruby-core - ml.ruby-lang.org

[ruby-core:114025] [Ruby master Feature#19744] Namespace on read
by tagomoris (Satoshi TAGOMORI) 06 May '25

06 May '25

Issue #19744 has been reported by tagomoris (Satoshi TAGOMORI). ---------------------------------------- Feature #19744: Namespace on read https://bugs.ruby-lang.org/issues/19744 * Author: tagomoris (Satoshi TAGOMORI) * Status: Open * Priority: Normal ---------------------------------------- # What is the "Namespace on read" This proposes a new feature to define virtual top-level namespaces in Ruby. Those namespaces can require/load libraries (either .rb or native extension) separately from the global namespace. Dependencies of required/loaded libraries are also required/loaded in the namespace. ### Motivation The "namespace on read" can solve the 2 problems below, and can make a path to solve another problem: The details of those motivations are described in the below section ("Motivation details"). #### Avoiding name conflicts between libraries Applications can require two different libraries safely which use the same module name. #### Avoiding unexpected globally shared modules/objects Applications can make an independent/unshared module instance. #### (In the future) Multiple versions of gems can be required Application developers will have fewer version conflicts between gem dependencies if rubygems/bundler will support the namespace on read. ### Example code with this feature ```ruby # your_module module YourModule end # my_module.rb require 'your_module' module MyModule end # example.rb namespace1 = NameSpace.new namespace1.require('my_module') #=> true namespace1::MyModule #=> #<Module:0x00000001027ea650>::MyModule (or #<NameSpace:0x00...>::MyModule ?) namespace1::YourModule # similar to the above MyModule # NameError YourModule # NameError namespace2 = NameSpace.new # Any number of namespaces can be defined namespace2.require('my_module') # Different library "instance" from namespace1 require 'my_module' # require in the global namespace MyModule.object_id != namespace1::MyModule.object_id #=> true namespace1::MyModule.object_id != namespace2::MyModule.object_id ``` The required/loaded libraries will define different "instances" of modules/classes in those namespaces (just like the "wrapper" 2nd argument of `Kernel.load`). This doesn't introduce compatibility problems if all libraries use relative name resolution (without forced top-level reference like `::Name`). # "On read": optional, user-driven feature "On read" is a key thing of this feature. That means: * No changes are required in existing/new libraries (except for limited cases, described below) * No changes are required in applications if it doesn't need namespaces * Users can enable/use namespaces just for limited code in the whole library/application Users can start using this feature step by step (if they want it) without any big jumps. ## Motivation details This feature can solve multiple problems I have in writing/executing Ruby code. Those are from the 3 problems I mentioned above: name conflicts, globally shared modules, and library version conflicts between dependencies. I'll describe 4 scenarios about those problems. ### Running multiple applications on a Ruby process Modern computers have many CPU cores and large memory spaces. We sometimes want to have many separate applications (either micro-service architecture or modular monolith). Currently, running those applications require different processes. It requires additional computation costs (especially in developing those applications). If we have isolated namespaces and can load applications in those namespaces, we'll be able to run apps on a process, with less overhead. (I want to run many AWS Lambda applications on a process in isolated namespaces.) ### Running tests in isolated namespaces Tests that require external libraries need many hacks to: * require a library multiple times * require many different 3rd party libraries into isolated spaces (those may conflict with each other) Software with plugin systems (for example, Fluentd) will get benefit from namespaces. In addition to it, application tests can avoid unexpected side effects if tests are executed in isolated namespaces. ### Safely isolated library instances Libraries may have globally shared states. For example, [Oj](https://github.com/ohler55/oj) has a global `Obj.default_options` object to change the library behavior. Those options may be changed by any dependency libraries or applications, and it changes the behavior of `Oj` globally, unexpectedly. For such libraries, we'll be able to instantiate a safe library instance in an isolated namespace. ### Avoiding dependency hells Modern applications use many libraries, and those libraries require much more dependencies. Those dependencies will cause version conflicts very often. In such cases, application developers should resolve those by updating each libraries, or should just wait for the new release of libraries to conflict those libraries. Sometimes, library maintainers don't release updated versions, and application developers can do nothing. If namespaces can require/load a library multiple times, it also enables to require/load different versions of a library in a process. It requires the support of rubygems, but namespaces should be a good fundamental of it. ## Expected problems ### Use of top-level references In my expectation, `::Name` should refer the top-level `Name` in the global namespace. I expect that `::ENV` should contain the environment variables. But it may cause compatibility problems if library code uses `::MyLibrary` to refer themselves in their deeply nested library code. ### Additional memory consumption An extension library (dynamically linked library) may be loaded multiple times (by `dlopen` for temporarily copied dll files) to load isolated library "instances" if different namespaces require the same extension library. That consumes additional memory. In my opinion, additional memory consumption is a minimum cost to realize loading extension libraries multiple times without compatibility issues. This occurs only when programmers use namespaces. And it's only about libraries that are used in 2 or more namespaces. ### The change of `dlopen` flag about extension libraries To load an extension library multiple times without conflicting symbols, all extensions should stop sharing symbols globally. Libraries referring symbols from other extension libraries will have to change code & dependencies. (About the things about extension libraries, [Naruse also wrote an entry](https://naruse.hateblo.jp/entry/2023/05/22/193411).) # Misc The proof-of-concept branch is here: https://github.com/tagomoris/ruby/pull/1 It's still work-in-progress branch, especially for extension libraries. -- https://bugs.ruby-lang.org/

17 70

[ruby-core:116370] [Ruby master Bug#20203] `TestEnumerable` test failures with GCC 14
by vo.x (Vit Ondruch) 28 Apr '25

28 Apr '25

Issue #20203 has been reported by vo.x (Vit Ondruch). ---------------------------------------- Bug #20203: `TestEnumerable` test failures with GCC 14 https://bugs.ruby-lang.org/issues/20203 * Author: vo.x (Vit Ondruch) * Status: Open * Priority: Normal * ruby -v: ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-linux] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- There is ongoing mass rebuild in Fedora and that is first time GCC 14 is used and we observe test failures in `TestEnumerable`. Here are a few examples: ~~~ [ 3000/26419] TestEnumerable#test_transient_heap_sort_bymalloc_consolidate(): unaligned fastbin chunk detected ~~~ ~~~ [ 2455/26535] TestEnumerable#test_transient_heap_sort_bycorrupted size vs. prev_size in fastbins ~~~ ~~~ [ 9716/26532] TestEnumerable#test_any_with_unused_blockdouble free or corruption (fasttop) ~~~ The full logs are accessible [here](https://koji.fedoraproject.org/koji/taskinfo?taskID=112176941). Please drill through `Descendants` and `build.log` -- https://bugs.ruby-lang.org/

5 12

[ruby-core:111526] [Ruby master Bug#19288] Ractor JSON parsing significantly slower than linear parsing
by maciej.mensfeld (Maciej Mensfeld) 18 Apr '25

18 Apr '25

Issue #19288 has been reported by maciej.mensfeld (Maciej Mensfeld). ---------------------------------------- Bug #19288: Ractor JSON parsing significantly slower than linear parsing https://bugs.ruby-lang.org/issues/19288 * Author: maciej.mensfeld (Maciej Mensfeld) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.0 (2022-12-25 revision a528908271) [x86_64-linux] * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- a simple benchmark: ```ruby require 'json' require 'benchmark' CONCURRENT = 5 RACTORS = true ELEMENTS = 100_000 data = CONCURRENT.times.map do ELEMENTS.times.map do { rand => rand, rand => rand, rand => rand, rand => rand }.to_json end end ractors = CONCURRENT.times.map do Ractor.new do Ractor.receive.each { JSON.parse(_1) } end end result = Benchmark.measure do if RACTORS CONCURRENT.times do |i| ractors[i].send(data[i], move: false) end ractors.each(&:take) else # Linear without any threads data.each do |piece| piece.each { JSON.parse(_1) } end end end puts result ``` Gives following results on my 8 core machine: ```shell # without ractors: 2.731748 0.003993 2.735741 ( 2.736349) # with ractors 12.580452 5.089802 17.670254 ( 5.209755) ``` I would expect Ractors not to be two times slower on the CPU intense work. -- https://bugs.ruby-lang.org/

9 21

[ruby-core:116555] [Ruby master Feature#20233] pkg-config not working on MSVC
by jmarrec (Julien Marrec) 13 Apr '25

13 Apr '25

Issue #20233 has been reported by jmarrec (Julien Marrec). ---------------------------------------- Feature #20233: pkg-config not working on MSVC https://bugs.ruby-lang.org/issues/20233 * Author: jmarrec (Julien Marrec) * Status: Open * Priority: Normal ---------------------------------------- The PKG_CONFIG variable is not respected in win32/Makefile.sub Also, need to ideally use the `--msvc-syntax` flag (pkgconf 1.4.0, released 7 years ago, adds it). But mostly, I realized that the `try_ldflags` in `lib/mkmf.rb` actually passes these ldflags as the `opt` parameter to the `link_command` function, and not as `ldflags`. While the UNIX systems are forgiving, the MSVC compiler is not, and because the flags are passed before the `-link`, MSVC complains that it ignores it and therefore cannot find the requested library. -- https://bugs.ruby-lang.org/

2 2

[ruby-core:116666] [Ruby master Bug#20255] Embedded arrays aren't moved correctly across ractors
by luke-gru (Luke Gruber) 07 Apr '25

07 Apr '25

Issue #20255 has been reported by luke-gru (Luke Gruber). ---------------------------------------- Bug #20255: Embedded arrays aren't moved correctly across ractors https://bugs.ruby-lang.org/issues/20255 * Author: luke-gru (Luke Gruber) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- `ractor.send(ary, move: true)` works incorrectly because if `ary` is embedded, the new moved object doesn't populate its own embedded space, it uses the MovedObject's embedded space. example: ```ruby r = Ractor.new { inner_ary = receive values = {} values[:equal] = (inner_ary == ["",{},2,3,4,5,6]) values[:string] = inner_ary.to_s values } ary = [String.new,Hash.new,2,3,4,5,6] r.send(ary, move: true) r_values = r.take p r_values[:equal] p r_values[:string] # => false # => "[\"\", {}, 2, 2.0, 21747991570, String, 3]" ``` -- https://bugs.ruby-lang.org/

3 17

[ruby-core:114410] [Ruby master Bug#19841] Marshal.dump stack overflow with recursive Time
by segiddins (Samuel Giddins) 29 Mar '25

29 Mar '25

Issue #19841 has been reported by segiddins (Samuel Giddins). ---------------------------------------- Bug #19841: Marshal.dump stack overflow with recursive Time https://bugs.ruby-lang.org/issues/19841 * Author: segiddins (Samuel Giddins) * Status: Open * Priority: Normal * ruby -v: 3.2.2 * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- ``` ruby #!/usr/bin/env ruby puts RUBY_VERSION t = Time.at(0, 1, :nanosecond) t.instance_variable_set :@itself, t Marshal.dump(t) ``` Yields a stack overflow error from the `Marshal.dump` call, even though Marshal is explicitly able to handle cyclical references -- https://bugs.ruby-lang.org/

6 8

[ruby-core:114147] [Ruby master Bug#19765] Ractor.make_shareable ignores self of a proc created from a Method
by Ethan (Ethan -) 27 Mar '25

27 Mar '25

Issue #19765 has been reported by Ethan (Ethan -). ---------------------------------------- Bug #19765: Ractor.make_shareable ignores self of a proc created from a Method https://bugs.ruby-lang.org/issues/19765 * Author: Ethan (Ethan -) * Status: Open * Priority: Normal * ruby -v: ruby 3.3.0dev (2023-07-12T00:26:03Z master dfe782be17) [x86_64-darwin21] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- An unshareable receiver of a Proc or a Method will cause make_shareable to error, but this does not happen with a proc from Method#to_proc: ```ruby str = "" a = str.instance_exec { proc { to_s } } Ractor.make_shareable a # => <internal:ractor>:820:in `make_shareable': Proc's self is not shareable: #<Proc:0x00000001064b62c8 (irb):1> (Ractor::IsolationError) b = str.instance_exec { method(:to_s) } Ractor.make_shareable b # => <internal:ractor>:820:in `make_shareable': can not make shareable object for #<Method: String#to_s()> (Ractor::Error) c = str.instance_exec { method(:to_s).to_proc } Ractor.make_shareable c c.call # => "" str[0] = "!" c.call # => "!" ``` Related, maybe: #19372 #19374 Tangential: why does Proc cause Ractor::IsolationError but Method causes Ractor::Error? -- https://bugs.ruby-lang.org/

3 2

[ruby-core:118823] [Ruby master Bug#20670] fork deadlocks in child process due to timer thread
by jhawthorn (John Hawthorn) 22 Mar '25

22 Mar '25

Issue #20670 has been reported by jhawthorn (John Hawthorn). ---------------------------------------- Bug #20670: fork deadlocks in child process due to timer thread https://bugs.ruby-lang.org/issues/20670 * Author: jhawthorn (John Hawthorn) * Status: Open * ruby -v: ruby 3.3.4 (2024-07-09 revision be1089c8ec) [x86_64-linux] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- We've been seeing an occasional failure in the Rails CI related to a test which forks and I managed to reduce it to the following reproduction. ``` ruby Thread.new do loop { sleep 0.0001 } end 1000.times do pid = fork{} Process.waitpid(pid) rescue Exception Process.kill(:KILL, pid) raise end ``` This hangs on Ruby 3.3 and HEAD (_fairly_ reliably), but completes always on Ruby 3.2 In a debugger it seems like the timer thread acquires `vm->ractor.sched.lock` in the parent process just as the process is forking. The child process then ends up stuck inside of `thread_sched_atfork` when trying to acquire the same lock. I've opened https://github.com/ruby/ruby/pull/11356 with a fix -- https://bugs.ruby-lang.org/

4 4

[ruby-core:114936] [Ruby master Feature#19908] Update to Unicode 15.1
by nobu (Nobuyoshi Nakada) 18 Mar '25

18 Mar '25

Issue #19908 has been reported by nobu (Nobuyoshi Nakada). ---------------------------------------- Feature #19908: Update to Unicode 15.1 https://bugs.ruby-lang.org/issues/19908 * Author: nobu (Nobuyoshi Nakada) * Status: Assigned * Priority: Normal * Assignee: duerst (Martin Dürst) * Target version: 3.3 ---------------------------------------- The Unicode 15.1 is released. The current enc-unicode.rb seems to fail because of `Indic_Conjunct_break` properties with values. I'm not sure how these properties should be handled well. `/\p{InCB_Liner}/` or `/\p{InCB=Liner}/` as the comments in that file? https://github.com/nobu/ruby/tree/unicode-15.1 is the former. -- https://bugs.ruby-lang.org/

14 23

[ruby-core:117188] [Ruby master Bug#20340] Ractor comments not applying to constant targets
by kddnewton (Kevin Newton) 13 Mar '25

13 Mar '25

Issue #20340 has been reported by kddnewton (Kevin Newton). ---------------------------------------- Bug #20340: Ractor comments not applying to constant targets https://bugs.ruby-lang.org/issues/20340 * Author: kddnewton (Kevin Newton) * Status: Open * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- In this example: ```ruby C, = { foo: bar } ``` You get: ``` == disasm: #<ISeq:<main>@test.rb:1 (1,0)-(1,17)> 0000 putobject :foo ( 1)[Li] 0002 putself 0003 opt_send_without_block <calldata!mid:bar, argc:0, FCALL|VCALL|ARGS_SIMPLE> 0005 newhash 2 0007 dup 0008 expandarray 1, 0 0011 putspecialobject 3 0013 setconstant :C 0015 leave ``` But there's no difference from: ```ruby # shareable_constant_value: literal C, = { foo: bar } ``` I would have expected: ``` == disasm: #<ISeq:<main>@test.rb:1 (1,0)-(2,16)> 0000 putobject RubyVM::FrozenCore ( 2)[Li] 0002 putobject :foo 0004 putobject RubyVM::FrozenCore 0006 putself 0007 opt_send_without_block <calldata!mid:bar, argc:0, FCALL|VCALL|ARGS_SIMPLE> 0009 putobject "C" 0011 opt_send_without_block <calldata!mid:ensure_shareable, argc:2, ARGS_SIMPLE> 0013 newhash 2 0015 opt_send_without_block <calldata!mid:make_shareable, argc:1, ARGS_SIMPLE> 0017 dup 0018 expandarray 1, 0 0021 putspecialobject 3 0023 setconstant :C 0025 leave ``` Is this omission intentional? For context, I'm building this in prism and I need to know if I should explicitly disable this behavior. -- https://bugs.ruby-lang.org/

2 2