October 2023 - ruby-core - ml.ruby-lang.org

[ruby-core:114422] [Ruby master Feature#19842] Intorduce M:N threads
by ko1 (Koichi Sasada) 10 Nov '23

10 Nov '23

Issue #19842 has been reported by ko1 (Koichi Sasada). ---------------------------------------- Feature #19842: Intorduce M:N threads https://bugs.ruby-lang.org/issues/19842 * Author: ko1 (Koichi Sasada) * Status: Open * Priority: Normal * Assignee: ko1 (Koichi Sasada) ---------------------------------------- This ticket proposes to introduce M:N threads to improve Threads/Ractors performance. ## Background Ruby threads (RT in short) are implemented from old Ruby versions and they have the following features: * Can be created with simple notation `Thread.new{}` * Can be switched to another ready Ruby thread by: * Time-slice. * I/O blocking. * Synchronization such as Mutex features. * And other blocking reasons. * Can be interruptible by: * OS-deliver signals (only for the main thread). * `Thread#kill`. * Can be terminated by: * the end of each Ruby thread. * the end of the main thread (and other Ruby threads are killed). Ruby 1.8 and erlier versions uses M:1 threads (green threads, user level threads, .... the word 1:N threads is more popular but to make this explanation consistent I use "M:1" term here) which manages multiple Ruby threads on 1 native thread. (Native threads are provided by C interfaces such as Pthreads. In many cases, native threads are OS threads, but there are also user-level implementations, such as user-level pthread libraries in theory. Therefore, they are referred to as native threads in this article and NT in short) If a Ruby thread T1 blocked because of a I/O operation, Ruby interpreter switches to the next ready Ruby thread T2. The I/O operation will be monitors by a `select()` (or similar) functionality and if the I/O is ready, T1 is marked as a ready thread and T1 will be resumed soon. However, when a Ruby thread issues some other blocking operations such as `gethostbyname()`, Ruby interpreter can not swtich to any other Ruby thread while `gethostbyname()` is not finished. We named two types blocking operations: * Managed blocking operations * I/O (most of read/write) * manage by I/O multiplexing API (select, poll, epoll, kqueue, IOCP, io_uring, ...) * Sleeping * Synchronization (Mutex, Queue, ...) * Unmanaged operations * All other blocking operations not listed above, written in C * Huge number calculation like `Bignum#*` * DNS lookup * I/O (can not detect block-able or not by multiplexing API) * open on FIFO, close on NFS, ... * flock and other locking mechanism * library call which uses blocking operations * `libfoo` has `foo_func()` and `foo_func()` waits DNS lookup. A Ruby extension `foo-ruby` can call `foo_func()`. With these terms we can say that M:1 threads can suport managed blocking operations but can not support unmanaged operations (can not make progress other Ruby threads) without further tricks. Note that if the `select()`-like system calls say a `fd` is ready, but the I/O opeartion for `fd` can be blocked because of some contention (read by another thread or process, for example). M:1 threads has another disadvantage that it can not run in parallel because only a native thread is used. From Ruby 1.9 we had implemented 1:1 thread which means a Ruby thread has a corresponding native thread. To make implementation easy we also introduced a GVL. Only a Ruby thread acquires GVL can run. With 1:1 model, we can support managed blocking oprations and unmanaged blocking operations by releasing GVL. When a Ruby thread want to issue a blocking operation, the Ruby thread releases GVL and another ready Ruby threads continue to run. We don't care the blocking operation is managed or unmanaged. (We can not make some of unmanaged blocking operations interruptible (stop by Ctrl-C for example)). Advantages of 1:1 threads to the M:1 threads is: * Easy to handle blocking operations by releasing GVL. * We can utilize parallelism with multiple native threads by releasing GVL. Disadvantages of 1:1 threads to the M:1 threads is: * Overhead to make many native threads for many Ruby threads * We can not make huge number of Ruby threads and Ractors on 1:1 threads. * Thread switching overhead by GVL because inter-core communication is needed. From Ruby 3.0 we introduced fiber scheduler mechanism to maintain multiple fibers Differences between Ruby 1.8 M:1 threads are: * No timeslice (only switch fibers by managed blocking operations) * Ruby users can make own schedulers for apps with favorite underlying mechanism Disadvantages are similar to M:1 threads. Another disadvantages is we need to consider about Fiber's behavior. From Ruby 3.0 we also introduced Ractors. Ractors can run in parallel because of separating most of objects. 1 Ractor creates 1 Ruby thread, so Ractors has same disadvantages of 1:1 threads. For example, we can not make huge number of Ractors. ## Goal Our goal is making lightweight Ractors on lightweight Ruby threads. To enable this goal we propose to implement M:N threads on MRI. M:N threads manages M Ruby threads on N native threads, with limited N (~= CPU core numbers for example). Advantages of M:N threads are: 1. We can run N ractors on N native threads simultaneously if the machine has N cores. 2. We can make huge number of Ruby threads or Ractors because we don't need huge number of native threads 3. We can support unmanaged blocking operations by locking a native thread to a Ruby thread which issues an unmanaged blocking operation. 4. We can make our own Ruby threads or Ractors scheduler instead of the native thread (OS) scheduler. Disadvantages of M:N threads are: 1. It is complex implmentation and it can be hard. 2. It can introduce incompatibility especaially on TLS (Thread local storage). 3. We need to maitain our own scheduler. Without using multiple Ractors, it is similar to Ruby 1.8 M:1 threads. The difference with M:1 threads are locking NT mechanism to support unmanaged blocking operations. Another advantage is that it is easy to fallback to 1:1 threads by locking all of corresponding native threads to Ruby threads. ## Proposed design ### User facing changes If a program only has a main Ractor (i.e., most Ruby programs), the user will not face any changes by default. On main Ractor, all threads are 1:1 threads by default and there is no compatibility issue. `RUBY_MN_THREADS=1` envrionment variable is given, main Ractor enables M:N threads. Note that the main thread locks NT by default because the initial NT is special in some case. I'm not sure we can relax this limitation. On the multiple Ractors, N (+ alpha) native threads run M ractors. Now there is no way to disable M:N threads on multiple Ractors because there are only a few multi-Ractor programs and no compatibility issues. Maximum number of N can be specified by `RUBY_MAX_PROC=N`. 8 by default but this value should be specified with the number of CPU processors (cores). ### TLS issue On M:N threads a Ruby thread (RT1) migrates from a native thread (NT1) to NT2, ... so that TLS on native code can be a problem. For example, RT1 calls a library function `foo()` and it set TLS1 on NT1. After migrating RT1 to NT2, RT1 calls `foo()` again but there is no TLS1 record because TLS1 is recorded only on NT1. On this case, RT1 should be run on NT1 while using native library foo. To avoid such prbolem, we need the following features: * 1:1 threads on main Ractor by default * functionality to lock the NT for RT, maybe `Thread#lock_native_thread` and `Thread#unlock_native_thread` API is needed. For example, Go language has `runtime.LockOSThread()` and `runtime.UnlockOSThread()` for this purpose. * Or C-API only for this purpose? (not fixed yet) Thankfully, the same problem can occur with Fiber scheduler (and of course Ruby 1.8 M:1 threads), but I have not heard of it being much of a problem, so I expect that TLS will not be much of an issue. ### Unmanaged blocking operations From Ruby 1.9 (1:1 threads), the `nogvl(func)` API is used for most blocking operations to keep the threading system healthy. In other words, `nogvl(func)` represents that the given function is blocking operation. To support unmanaged blocking operations, we lock a native thread for the Ruby thread which issues blocking operation. If the blocking operations doesn't finish soon, other Ruby threads can not run because a RT locks NT. In this case, another system monitoring thread named "Timer thread" (historical name and TT in short) creates another NT to run ready other Ruby threads. This TT's behavior is the same as the behavior of "sysmon" in the Go language. We named locked NT as dedicated native threads (DNT) and other NT as shared native threads (SNT). The upper bound by `RUBY_MAX_PROC` affects the number of SNT. In other words, the number of DNT is not limited (it is same that the number of NT on 1:1 threads are not limited). ### Managed blocking operations Managed blocking operations are multiplexing by `select()`-like functions on the Timer thread.. Now only `epoll()` is supported. I/O operation flow (read on fd1) on Ruby thread RT1: 1. check the ready-ness of fd1 by `poll(timeout = 0)`, goto step 4. 2. register fd1 to Timer thread (TT) epoll and resume another ready Ruby thread. 3. If TT detects that the fd1 is ready, make RT1 as ready thread. 4. When RT1 is resumed, then do `read()` by locking corresponding NT1. `sleep(n)` operation flow on Ruby thread RT1: 1. register timeout of RT1 to TT epoll. 2. If TT detects the timeout of RT1 (n seconds), TT makes RT1 as a ready Ruby thread. ### Internal design * 2 level scheduling * Ruby threads of a Ractor is managed by 1:N threads * Ruby threads of different Ractors are managed by M:N threads * Timer thread has several duties 1. Monitoring I/O (or other event) ready-ness 2. Monitoring timeout 3. Produce timeslice signals 4. Help OS signal delivering (On pthread environment) recent Ruby doesn't make timer thread but MaNy implementation makes TT anytime. it can be improved. ## Implementation The code name is MaNy project, it is from MN threads. https://github.com/ko1/ruby/tree/many2 The implementation is not matured (debugging now). ## Measurements See RubyKaigi 2023 slides: https://atdot.net/~ko1/activities/2023_rubykaigi2023.pdf ## Discussion * Enable/disable * default behavior * how to switch the behavior * Should we lock the NT for main thread anytime? * Ruby/C API to lock the native threads ## Misc This description will be improved more later. -- https://bugs.ruby-lang.org/

7 29

[ruby-core:114985] [Ruby master Bug#19916] URI#to_s can serialize to a value that doesn't deserialize to the original
by yawboakye (yaw boakye) 09 Nov '23

09 Nov '23

Issue #19916 has been reported by yawboakye (yaw boakye). ---------------------------------------- Bug #19916: URI#to_s can serialize to a value that doesn't deserialize to the original https://bugs.ruby-lang.org/issues/19916 * Author: yawboakye (yaw boakye) * Status: Open * Priority: Normal * ruby -v: 3.2.2 * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- It appears that when we're serializing a URI to string, we don't check/confirm that it is represented in a form that can be deserialized back into the original. I think it's fair to expect that serialize+deserialize produces an object that is the same as the original, only differing perhaps in terms of the unique object identifier. This isn't the case with URI when they are custom built, which might happen a lot, for example in a Rails app that accepts URL inputs from users. Let me attempt a reproduction, using the generic URI `example.com`. ```ruby example_url = "example.com" example_uri = URI(example_url) ``` Given that no scheme is explicitly set in the URI, it is correctly parsed as generic, with the given `example.com` interpreted as the path. The object returned to is mutable. Since we didn't automatically detect a scheme, let's fix that as well as the hostname. ``` ruby example_uri.scheme = "https" example_uri.hostname = example_uri.path # I've intentionally left path value unchanged, since it helps demonstrate the potential bug. ``` Given that we have a scheme, an authority, and a path, and given that we format URI according to [RFC 3986], one may expect that serializing the URI to string will follow the guidelines of section 3 of the RFC: [Syntax Components], which requires a slash separator between the authority (in our case hostname) and the path. It appears that `URI#to_s` may not do that if path didn't already have a slash prefix. Which would be fine if we were keeping an invariant that ensured that we never produced bad serialized URI. To return to our `example_uri`, serialization produces: ```ruby serialized_uri = example_uri.to_s puts serialized_uri # https://example.comexample.com ``` This is obviously bad. One would have expected `https://example.com/example.com` instead. That is, the slash will be automatically and correctly inserted, just as the double slashes were automatically inserted between the scheme and and the authority. `serialized_uri` cannot be deserialized into `example_uri`, in fact. Below is an attempt at deserialization and a comparison of the new value to the original: ```ruby deserialized_example_uri = URI(serialized_uri) example_uri.scheme == deserialized_example_uri.scheme # true example_uri.hostname == deserialized_example_uri.hostname # false (for, example.com =/= example.comexample.com) example_uri.path == deserialized_example_uri.path # false (for, example.com =/= "") ``` I believe that the ability to serialize and deserialize an object without losing fidelity is a great thing. I believe even more strongly that we should preserve/maintain an invariant that allows us to always serialize a URI to a format that meets the RFC's specification. Therefore I consider this a bug, and I'd be willing to work on a fix, as my first contribution to Ruby, if enough people consider it a bug too. Regards! [RFC 3986]: https://www.rfc-editor.org/rfc/rfc3986 [Syntax Components]: https://www.rfc-editor.org/rfc/rfc3986#section-3 ---Files-------------------------------- Screenshot 2023-09-29 at 12.19.26.png (180 KB) -- https://bugs.ruby-lang.org/

4 3

[ruby-core:115060] [Ruby master Bug#19928] IPV6 nameserver issues: "Protocol not supported - socket(2) - udp"
by brendans-bits (Brendan Bits) 07 Nov '23

07 Nov '23

Issue #19928 has been reported by brendans-bits (Brendan Bits). ---------------------------------------- Bug #19928: IPV6 nameserver issues: "Protocol not supported - socket(2) - udp" https://bugs.ruby-lang.org/issues/19928 * Author: brendans-bits (Brendan Bits) * Status: Open * Priority: Normal * ruby -v: ruby 3.1.4p223 (2023-03-30 revision 957bb7cb81) [amd64-freebsd13] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- I get a similar error message to https://bugs.ruby-lang.org/issues/9477 The error message I see: ``` Errno::EPROTONOSUPPORT: Protocol not supported - socket(2) - udp ``` Similarly: ``` $ ruby -rresolv -e 'puts Resolv::DNS.new.getresource("_rubygems._tcp.rubygems.org", Resolv::DNS::Resource::IN::SRV).target' /usr/local/lib/ruby/3.1/resolv.rb:750:in `initialize': Protocol not supported - socket(2) - udp (Errno::EPROTONOSUPPORT) from /usr/local/lib/ruby/3.1/resolv.rb:750:in `new' from /usr/local/lib/ruby/3.1/resolv.rb:750:in `block (2 levels) in lazy_initialize' from /usr/local/lib/ruby/3.1/resolv.rb:740:in `each' from /usr/local/lib/ruby/3.1/resolv.rb:740:in `block in lazy_initialize' from /usr/local/lib/ruby/3.1/resolv.rb:735:in `synchronize' from /usr/local/lib/ruby/3.1/resolv.rb:735:in `lazy_initialize' from /usr/local/lib/ruby/3.1/resolv.rb:771:in `sender' from /usr/local/lib/ruby/3.1/resolv.rb:527:in `block in fetch_resource' from /usr/local/lib/ruby/3.1/resolv.rb:1125:in `block (3 levels) in resolv' from /usr/local/lib/ruby/3.1/resolv.rb:1123:in `each' from /usr/local/lib/ruby/3.1/resolv.rb:1123:in `block (2 levels) in resolv' from /usr/local/lib/ruby/3.1/resolv.rb:1122:in `each' from /usr/local/lib/ruby/3.1/resolv.rb:1122:in `block in resolv' from /usr/local/lib/ruby/3.1/resolv.rb:1120:in `each' from /usr/local/lib/ruby/3.1/resolv.rb:1120:in `resolv' from /usr/local/lib/ruby/3.1/resolv.rb:521:in `fetch_resource' from /usr/local/lib/ruby/3.1/resolv.rb:507:in `each_resource' from /usr/local/lib/ruby/3.1/resolv.rb:488:in `getresource' from -e:1:in `<main>' ``` I am running this in a FreeBSD jail (created by Bastille) that does not have IPV6 connectivity, on a host that does have IPV6 connectivity. Similar to the previous issue, the problem is successfully resolved by commenting out the IPV6 nameserver lines. Longer output for reference: ``` rake aborted! Errno::EPROTONOSUPPORT: Protocol not supported - socket(2) - udp /usr/local/www/mastodon/app/validators/email_mx_validator.rb:49:in `block in resolve_mx' /usr/local/www/mastodon/app/validators/email_mx_validator.rb:46:in `resolve_mx' /usr/local/www/mastodon/app/validators/email_mx_validator.rb:14:in `validate' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:423:in `block in make_lambda' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:180:in `block (2 levels) in halting_and_conditional' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:687:in `block (2 levels) in default_terminator' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:686:in `catch' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:686:in `block in default_terminator' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:181:in `block in halting_and_conditional' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:595:in `block in invoke_before' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:595:in `each' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:595:in `invoke_before' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:106:in `run_callbacks' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:929:in `_run_validate_callbacks' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activemodel-7.0.8/lib/active_model/validations.rb:406:in `run_validations!' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activemodel-7.0.8/lib/active_model/validations/callbacks.rb:115:in `block in run_validations!' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:107:in `run_callbacks' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:929:in `_run_validation_callbacks' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activemodel-7.0.8/lib/active_model/validations/callbacks.rb:115:in `run_validations!' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activemodel-7.0.8/lib/active_model/validations.rb:337:in `valid?' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activerecord-7.0.8/lib/active_record/validations.rb:68:in `valid?' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activerecord-7.0.8/lib/active_record/validations.rb:84:in `perform_validations' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activerecord-7.0.8/lib/active_record/validations.rb:47:in `save' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activerecord-7.0.8/lib/active_record/transactions.rb:298:in `block in save' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activerecord-7.0.8/lib/active_record/transactions.rb:354:in `block in with_transaction_returning_status' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activerecord-7.0.8/lib/active_record/connection_adapters/abstract/database_statements.rb:314:in `transaction' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activerecord-7.0.8/lib/active_record/transactions.rb:350:in `with_transaction_returning_status' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activerecord-7.0.8/lib/active_record/transactions.rb:298:in `save' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activerecord-7.0.8/lib/active_record/suppressor.rb:50:in `save' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activerecord-7.0.8/lib/active_record/autosave_association.rb:455:in `save_has_one_association' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activerecord-7.0.8/lib/active_record/autosave_association.rb:198:in `block in add_autosave_association_callbacks' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activerecord-7.0.8/lib/active_record/autosave_association.rb:166:in `instance_eval' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activerecord-7.0.8/lib/active_record/autosave_association.rb:166:in `block in define_non_cyclic_method' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:400:in `block in make_lambda' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:236:in `block in halting_and_conditional' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:599:in `block in invoke_after' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:599:in `each' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:599:in `invoke_after' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:108:in `run_callbacks' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:929:in `_run_create_callbacks' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activerecord-7.0.8/lib/active_record/callbacks.rb:459:in `_create_record' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activerecord-7.0.8/lib/active_record/timestamp.rb:108:in `_create_record' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activerecord-7.0.8/lib/active_record/persistence.rb:1069:in `create_or_update' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activerecord-7.0.8/lib/active_record/callbacks.rb:455:in `block in create_or_update' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:118:in `block in run_callbacks' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activerecord-7.0.8/lib/active_record/autosave_association.rb:370:in `around_save_collection_association' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:127:in `block in run_callbacks' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:138:in `run_callbacks' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:929:in `_run_save_callbacks' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activerecord-7.0.8/lib/active_record/callbacks.rb:455:in `create_or_update' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activerecord-7.0.8/lib/active_record/timestamp.rb:126:in `create_or_update' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activerecord-7.0.8/lib/active_record/persistence.rb:615:in `save' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activerecord-7.0.8/lib/active_record/validations.rb:47:in `save' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activerecord-7.0.8/lib/active_record/transactions.rb:298:in `block in save' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activerecord-7.0.8/lib/active_record/transactions.rb:354:in `block in with_transaction_returning_status' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activerecord-7.0.8/lib/active_record/connection_adapters/abstract/database_statements.rb:314:in `transaction' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activerecord-7.0.8/lib/active_record/transactions.rb:350:in `with_transaction_returning_status' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activerecord-7.0.8/lib/active_record/transactions.rb:298:in `save' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activerecord-7.0.8/lib/active_record/suppressor.rb:50:in `save' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activerecord-7.0.8/lib/active_record/autosave_association.rb:491:in `save_belongs_to_association' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activerecord-7.0.8/lib/active_record/autosave_association.rb:210:in `block in add_autosave_association_callbacks' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activerecord-7.0.8/lib/active_record/autosave_association.rb:166:in `instance_eval' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activerecord-7.0.8/lib/active_record/autosave_association.rb:166:in `block in define_non_cyclic_method' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:400:in `block in make_lambda' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:199:in `block (2 levels) in halting' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:687:in `block (2 levels) in default_terminator' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:686:in `catch' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:686:in `block in default_terminator' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:200:in `block in halting' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:595:in `block in invoke_before' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:595:in `each' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:595:in `invoke_before' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:116:in `block in run_callbacks' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:138:in `run_callbacks' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:929:in `_run_save_callbacks' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activerecord-7.0.8/lib/active_record/callbacks.rb:455:in `create_or_update' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activerecord-7.0.8/lib/active_record/timestamp.rb:126:in `create_or_update' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activerecord-7.0.8/lib/active_record/persistence.rb:615:in `save' # Generated by resolvconf /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activerecord-7.0.8/lib/active_record/validations.rb:47:in `save' # Generated by resolvconf /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activerecord-7.0.8/lib/active_record/transactions.rb:298:in `block in save' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activerecord-7.0.8/lib/active_record/transactions.rb:354:in `block in with_transaction_returning_status' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activerecord-7.0.8/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activesupport-7.0.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activesupport-7.0.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activesupport-7.0.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activesupport-7.0.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activerecord-7.0.8/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activerecord-7.0.8/lib/active_record/connection_adapters/abstract/database_statements.rb:316:in `transaction' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activerecord-7.0.8/lib/active_record/transactions.rb:350:in `with_transaction_returning_status' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activerecord-7.0.8/lib/active_record/transactions.rb:298:in `save' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/activerecord-7.0.8/lib/active_record/suppressor.rb:50:in `save' /usr/local/www/mastodon/lib/tasks/mastodon.rake:517:in `block (2 levels) in <main>' /usr/local/www/mastodon/vendor/bundle/ruby/3.1/gems/rake-13.0.6/exe/rake:27:in `<top (required)>' Tasks: TOP => mastodon:setup (See full trace by running task with --trace) ``` -- https://bugs.ruby-lang.org/

2 1

[ruby-core:115033] [Ruby master Misc#19925] DevMeeting-2023-11-07
by mame (Yusuke Endoh) 06 Nov '23

06 Nov '23

Issue #19925 has been reported by mame (Yusuke Endoh). ---------------------------------------- Misc #19925: DevMeeting-2023-11-07 https://bugs.ruby-lang.org/issues/19925 * Author: mame (Yusuke Endoh) * Status: Open * Priority: Normal ---------------------------------------- # The next dev meeting **Date: 2023/11/07 13:00-17:00** (JST) Log: *TBD* - Dev meeting *IS NOT* a decision-making place. All decisions should be done at the bug tracker. - Dev meeting is a place we can ask Matz, nobu, nurse and other developers directly. - Matz is a very busy person. Take this opportunity to ask him. If you can not attend, other attendees can ask instead of you (if attendees can understand your issue). - We will write a record of the discussion in the file or to each ticket in English. - All activities are best-effort (keep in mind that most of us are volunteer developers). - The date, time and place of the meeting are scheduled according to when/where we can reserve Matz's time. - *DO NOT* discuss then on this ticket, please. # Call for agenda items If you have a ticket that you want matz and committers to discuss, please post it into this ticket in the following format: ``` * [Ticket ref] Ticket title (your name) * Comment (A summary of the ticket, why you put this ticket here, what point should be discussed, etc.) ``` Example: ``` * [Feature #14609] `Kernel#p` without args shows the receiver (ko1) * I feel this feature is very useful and some people say :+1: so let discuss this feature. ``` - It is recommended to add a comment by 2023/11/04. We hold a preparatory meeting to create an agenda a few days before the dev-meeting. - The format is strict. We'll use [this script to automatically create an markdown-style agenda](https://gist.github.com/mame/b0390509ce1491b43610b9ebb665eb86). We may ignore a comment that does not follow the format. - Your comment is mandatory. We cannot read all discussion of the ticket in a limited time. We appreciate it if you could write a short summary and update from a previous discussion. -- https://bugs.ruby-lang.org/

10 11

[ruby-core:114072] [Ruby master Bug#19754] `IO::Buffer#get_string` raises unsuitable exception for too large offset
by nobu (Nobuyoshi Nakada) 06 Nov '23

06 Nov '23

Issue #19754 has been reported by nobu (Nobuyoshi Nakada). ---------------------------------------- Bug #19754: `IO::Buffer#get_string` raises unsuitable exception for too large offset https://bugs.ruby-lang.org/issues/19754 * Author: nobu (Nobuyoshi Nakada) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- The message seems suitable when both of `offset` and `length` are given. ```ruby IO::Buffer.for("test").get_string(5, 0) #=> Specified offset+length exceeds buffer size! (ArgumentError) ``` But without `length`, it doesn't seem to check `offset` properly. ```ruby IO::Buffer.for("test").get_string(5) #=> negative string size (or size too big) (ArgumentError) ``` -- https://bugs.ruby-lang.org/

4 6

[ruby-core:115187] [Ruby master Bug#19976] test/fiber/test_queue.rb stuck tests in Ubuntu ppc64le
by jaruga (Jun Aruga) 30 Oct '23

30 Oct '23

Issue #19976 has been reported by jaruga (Jun Aruga). ---------------------------------------- Bug #19976: test/fiber/test_queue.rb stuck tests in Ubuntu ppc64le https://bugs.ruby-lang.org/issues/19976 * Author: jaruga (Jun Aruga) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- I have seen the `test_pop_with_timeout` and `test_pop_with_timeout_and_value` are stuck/hang on GCC compilers on ruby's master branch in RubyCI ppc64le server Focal/Jammy and Travis CI ppc64le. This ticket is to manage this issue. On August 27 2023, we saw the following stuck/hang issue in Travis ppc64le Ubuntu focal. The used GCC version was 10.5.0. https://app.travis-ci.com/github/ruby/ruby/jobs/608696247#L2355 ``` Retrying hung up testcases... [1/2] TestFiberQueue#test_pop_with_timeout_and_value = 0.00 s [2/2] TestFiberQueue#test_pop_with_timeout ====[ 540 seconds still running ]==== ====[ 1080 seconds still running ]==== ====[ 1620 seconds still running ]==== ====[ 2160 seconds still running ]==== ``` We upgraded RubyCI's ppc64le server from focal to jammy, and started to use newer GCC 11.4.0 (gcc-11) on the server. https://packages.ubuntu.com/jammy-updates/gcc-11 - 11.4.0 We have not seen the issue in the server after starting using the gcc 11.4.0. http://rubyci.s3.amazonaws.com/ppc64le/ruby-master/recent.html However, I saw this issue on October 27 2023 again in Travis Ubuntu ppc64le jammy when I tried to upgrade Travis ppc64le from focal to jammy. I didn't seen the issue in Travis ppc64le focal. The used gcc version is also 11.4.0. https://github.com/ruby/ruby/pull/8739 https://app.travis-ci.com/github/junaruga/ruby/jobs/612361931#L2930 ``` [1/2] TestFiberQueue#test_pop_with_timeout====[ 1080 seconds still running ]==== ====[ 1620 seconds still running ]==== ====[ 2160 seconds still running ]==== ``` This means something is different between RubyCI ppc64le server and Travis ppc64le environments for running the tests. I was able to reproduce this stuck/hang issue with the reproducing script below in RubyCI's ppc64le Ubuntu jammy server. https://github.com/junaruga/report-ruby-fiber-hung_up-tests The possible differences that may cause the issue is a parallel execution `make -jN`, or compiler flag`-O1`, or `-ggdb3`. https://github.com/junaruga/report-ruby-fiber-hung_up-tests/blob/d94205d9d7… I also sent the PR to make the stuck/hang tests fail, and it was merged. The tests failing immediately is better than the tests being stuck/hang. https://github.com/ruby/ruby/pull/8791 I hope we find the cause and fix this stuck/hang issue in gcc 11.4.0 in Ubuntu jammy. -- https://bugs.ruby-lang.org/

1 1

[ruby-core:115195] [Ruby master Bug#18743] Enumerator#next / peek re-use each others stacktraces
by marcper (Marcelo Pereira) 29 Oct '23

29 Oct '23

Issue #18743 has been updated by marcper (Marcelo Pereira). Hi @nobu, and @ko1. The change was merged in July. Shouldn't this issue be closed? ---------------------------------------- Bug #18743: Enumerator#next / peek re-use each others stacktraces https://bugs.ruby-lang.org/issues/18743#change-105109 * Author: sos4nt (Stefan Schüßler) * Status: Open * Priority: Normal * Assignee: ko1 (Koichi Sasada) * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN ---------------------------------------- I encountered an odd behavior. If I rescue the `StopIteration` exception from `peek` and call `next` afterwards: (or vice-versa) ```ruby # enum.rb # 1 # 2 enum = [].each # 3 enum.peek rescue nil # 4 enum.next # 5 ``` it will show the stacktrace from the rescued `peek` call: ``` $ ruby enum.rb enum.rb:4:in `peek': iteration reached an end (StopIteration) from enum.rb:4:in `<main>' ``` Whereas the error should refer to `next` on line number 5. The same happens when calling `peek` after `next` or when having muliple `peek` / `next` calls: ```ruby # enum.rb # 1 # 2 enum = [].each # 3 enum.peek rescue nil # 4 enum.next rescue nil # 5 enum.peek rescue nil # 6 puts "line #{__LINE__}" # 7 enum.next # 8 ``` The stacktrace from the first (rescued) `peek` or `next` call will be shown which doesn't reflect the actual error location: ``` $ ruby enum.rb line 7 enum.rb:4:in `peek': iteration reached an end (StopIteration) from enum.rb:4:in `<main>' ``` This is very confusing when debugging code. ---Files-------------------------------- 01-Recreate-stacktrace-enumerator.patch (1.29 KB) -- https://bugs.ruby-lang.org/

1 0

[ruby-core:115190] [Ruby master Feature#19978] NoMethodError and large objects should not create unwieldy error messages
by ConorOBR (Conor O'Brien) 29 Oct '23

29 Oct '23

Issue #19978 has been reported by ConorOBR (Conor O'Brien). ---------------------------------------- Feature #19978: NoMethodError and large objects should not create unwieldy error messages https://bugs.ruby-lang.org/issues/19978 * Author: ConorOBR (Conor O'Brien) * Status: Open * Priority: Normal ---------------------------------------- # Description Currently, in the error message for `NoMethodError`, the object in question is inspected and displayed to the user. This results in unwieldy error messages for large objects. This comes up most often for me when working with large hashes, but in principal it affects any object with an `inspect` method which can produce large output. # Reproduce There are many ways to reproduce this, but here is one. (Scales with the size of the object, so this gets out of hand quickly with, say, `1e7` as the upperbound.) ```rb (1..1e3).to_a/=3 ``` # `ruby -v` ``` ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x64-mingw-ucrt] ``` # Desired Behavior Some kind of elision, e.g., for the above, the desired output could be something like ```rb -e:1:in `<main>': undefined method `/' for [1, 2, 3, ..., 998, 999, 1000]:Array (NoMethodError) (1..1e3).to_a/=3 ^ ``` At the very least, not showing the entire object (even if that means showing none of it) if the `inspect` exceeds a certain length would be preferable. -- https://bugs.ruby-lang.org/

3 3

[ruby-core:115184] [Ruby master Bug#18886] Struct aref and aset don't trigger any tracepoints.
by jeremyevans0 (Jeremy Evans) 27 Oct '23

27 Oct '23

Issue #18886 has been updated by jeremyevans0 (Jeremy Evans). I submitted a pull request to fix this: https://github.com/ruby/ruby/pull/8782 ---------------------------------------- Bug #18886: Struct aref and aset don't trigger any tracepoints. https://bugs.ruby-lang.org/issues/18886#change-105099 * Author: ioquatix (Samuel Williams) * Status: Open * Priority: Normal * Assignee: ko1 (Koichi Sasada) * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN ---------------------------------------- Given the following program, `thing.name` and `thing.shape` don't trigger `c_call` trace points (or any trace points actually). ```ruby pp RUBY_VERSION trace_point = TracePoint.new(:line, :call, :c_call, :a_call) do |trace| puts trace.event if trace.event == :call # Ruby doesn't always mark call-sites in sub-expressions, so we use this approach to compute a call site and mark it: if location = caller_locations(2, 1).first and path = location.path puts "> #{path}:#{location.lineno}:#{trace.event}" end end if path = trace.path puts "= #{path}:#{trace.lineno}:#{trace.event}" end end trace_point.enable # This will trigger call trace points class Thing def name :cat end def shape :square end end thing = Thing.new # Thing = Struct.new(:name, :shape) # thing = Thing.new(:cat, :rectangle) [ name: thing.name, shape: thing.shape, ] ``` ## Current HEAD ``` = ../test.rb:30:line: = ../test.rb:30:c_call:new = ../test.rb:30:c_call:inherited = ../test.rb:30:c_call:singleton_method_added = ../test.rb:30:c_call:singleton_method_added = ../test.rb:30:c_call:singleton_method_added = ../test.rb:30:c_call:singleton_method_added = ../test.rb:30:c_call:singleton_method_added = ../test.rb:30:c_call:method_added = ../test.rb:30:c_call:method_added = ../test.rb:30:c_call:method_added = ../test.rb:30:c_call:method_added = ../test.rb:30:c_call:const_added = ../test.rb:31:line: = ../test.rb:31:c_call:new = ../test.rb:31:c_call:initialize = ../test.rb:34:line: ``` ## Proposed PR ``` = ../test.rb:30:line: = ../test.rb:30:c_call:new = ../test.rb:30:c_call:inherited = ../test.rb:30:c_call:singleton_method_added = ../test.rb:30:c_call:singleton_method_added = ../test.rb:30:c_call:singleton_method_added = ../test.rb:30:c_call:singleton_method_added = ../test.rb:30:c_call:singleton_method_added = ../test.rb:30:c_call:method_added = ../test.rb:30:c_call:method_added = ../test.rb:30:c_call:method_added = ../test.rb:30:c_call:method_added = ../test.rb:30:c_call:const_added = ../test.rb:31:line: = ../test.rb:31:c_call:new = ../test.rb:31:c_call:initialize = ../test.rb:34:line: = ../test.rb:34:c_call:name = ../test.rb:35:c_call:shape ``` The reason is the internal implementation of struct doesn't have trace point instrumentation in `vm_call_opt_struct_aset` or `vm_call_opt_struct_aref`. Proposed fix: https://github.com/ruby/ruby/pull/6071 but this would need a review, maybe @jeremyevans0 and @ko1 can help. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:115179] [Ruby master Bug#19974] OpenSSL::PKCS7 generates SegFault when parsing invalid data
by dprater (David Prater) 26 Oct '23

26 Oct '23

Issue #19974 has been reported by dprater (David Prater). ---------------------------------------- Bug #19974: OpenSSL::PKCS7 generates SegFault when parsing invalid data https://bugs.ruby-lang.org/issues/19974 * Author: dprater (David Prater) * Status: Open * Priority: Normal * ruby -v: ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [x86_64-linux-musl], ruby 3.1.3p185 (2022-11-24 revision 1a6b16756e) [arm64-darwin21] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- We ran into an instance of attempting to parse invalid PKCS7 certs that consistently causes a segfault in multiple versions of Ruby 3, on multiple platforms (Linux, Mac). To reproduce: ------- (within pry or irb or whatever ruby shell you prefer) ---------- data = "-----BEGIN PKCS7-----\nMAsGCSqGSIb3DQEHAg==\n-----END PKCS7-----\n" OpenSSL::PKCS7.new(data).certificates ----------------- This causes a segfault 100% of the time for me on both an M1 Mac as well as in docker containers running on both Mac as well as Linux hosts. The issue appears to be attempting to parse empty signedData. https://lapo.it/asn1js/#MAsGCSqGSIb3DQEHAg shows that this data contains the necessary ASN1 sequence to look like a valid PKCS7 certificate. However, the signedData is empty. I assume this is the cause of the issue, but I surely don't know. I've attached the info from the core dump inside a linux container running on an M1 Mac. ---Files-------------------------------- core_dump (145 KB) -- https://bugs.ruby-lang.org/

2 3