Issue #20197 has been updated by kjtsanaktsidis (KJ Tsanaktsidis). Assignee set to kjtsanaktsidis (KJ Tsanaktsidis) I hacked at the postponed job infrastructure pretty extensively in 3.3, I’ll have a look at this bug in the next few days. Thank you for your report! ---------------------------------------- Bug #20197: Postponed job invocations are significantly reduced in Ruby 3.3 https://bugs.ruby-lang.org/issues/20197#change-106371 * Author: osyoyu (Daisuke Aritomo) * Status: Open * Priority: Normal * Assignee: kjtsanaktsidis (KJ Tsanaktsidis) * ruby -v: ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-linux] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- The number of postponed job invocations has been significantly reduced in Ruby 3.3. While my understanding is that postponed jobs provide no guarantee of how soon registered callbacks will fire, I believe the current rate is too low for practical usage, especially for profilers such as StackProf. A git bisect led me to https://github.com/ruby/ruby/commit/1f0304218cf00e05a4a126196676ba221ebf91f6 which obviously seems to be related, but I'm not sure why. ## Repro ### Expected The job fires (nearly) every 100 ms. ``` % ruby bin/test.rb # runs for 3 seconds count: 1 count: 2 (snip) count: 29 ``` ### Actual The job fires only once. ``` % ruby bin/test.rb count: 1 count: 2 (snip) count: 29 ``` ### Code ```ruby require 'mycext' time = Time.now th = Thread.new do loop do sleep 0.01 break if Time.now - time > 3 # run for 3 seconds end end th.join ``` ```c #include <pthread.h> #include <stdio.h> #include <time.h> #include "ruby.h" #include "ruby/debug.h" int called_count; void postponed_job(void *ptr) { called_count++; printf("count: %d\n", called_count); } _Noreturn void * pthread_main(void *_) { while (1) { rb_postponed_job_register_one(0, postponed_job, NULL); // Sleep for 100 ms struct timespec ts; ts.tv_sec = 0; ts.tv_nsec = 100 * 1000 * 1000; nanosleep(&ts, NULL); } } RUBY_FUNC_EXPORTED void Init_mycext(void) { called_count = 0; pthread_t pthread; pthread_create(&pthread, NULL, pthread_main, NULL); } ``` -- https://bugs.ruby-lang.org/

Issue #20197 has been updated by kjtsanaktsidis (KJ Tsanaktsidis). OK, I investigated this a bit. The reason your example only runs the callback once is that the "is-an-interrupt-pending?" flag is a per-thread (technically per execution context) state. Normally, the `rb_postponed_job_register_one` (and the new `rb_postponed_job_trigger` function as well) will set the interrupt-pending flag on the currently thread, which is what you want when running in a signal handler. However, when you're running from a non-ruby thread, you need to pick a ruby thread to set the flag on. The previous implementation before https://github.com/ruby/ruby/commit/1f0304218cf00e05a4a126196676ba221ebf91f6 would pick `vm->ractor.main_ractor->threads.running_ec`, which is actually the previous thread to run if there is no currently running thread. Now, however, we always pick the main thread (this seems to be because the semantics of `running_ec` are different when using the new M:N threading model). Your example has the main thread sleeping, so it will never check the interrupt-pending flag. Thus, now it doesn't run the callbacks (until the other thread is done). This PR will I think fix the issue with your reproduction: https://github.com/ruby/ruby/pull/9633. Are you able to share some more details about your use-case for postponed_job by the way? I ask, because even with this fix (and restoring the Ruby 3.2 behaviour), there are still some race conditions possible (e.g. `running_ec` might have _just_ entered a long sleep). If we want to make it possible to robustly run postponed jobs "soon" from non-Ruby threads, I think we would need: * Add a VM-global interrupt flag * Have `rb_thread_check_ints` check the VM-global flag as well as its own EC-local flag, * Have `rb_postponed_job_trigger` see if we're running on a Ruby thread, and if not, set the VM-global interrupt flag instead of the EC-local one. This seems like a fair bit of extra complexity to carry around though if there isn't a use-case which requires it. ---------------------------------------- Bug #20197: Postponed job invocations are significantly reduced in Ruby 3.3 https://bugs.ruby-lang.org/issues/20197#change-106382 * Author: osyoyu (Daisuke Aritomo) * Status: Open * Priority: Normal * Assignee: kjtsanaktsidis (KJ Tsanaktsidis) * ruby -v: ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-linux] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- The number of postponed job invocations has been significantly reduced in Ruby 3.3. While my understanding is that postponed jobs provide no guarantee of how soon registered callbacks will fire, I believe the current rate is too low for practical usage, especially for profilers such as StackProf. A git bisect led me to https://github.com/ruby/ruby/commit/1f0304218cf00e05a4a126196676ba221ebf91f6 which obviously seems to be related, but I'm not sure why. ## Repro ### Expected The job fires (nearly) immediately after being triggered. In the following example, the job is triggered every 100 ms. ``` % ruby bin/test.rb # runs for 3 seconds count: 1 count: 2 (snip) count: 29 ``` ### Actual The job gets fired only once. ``` % ruby bin/test.rb count: 1 ``` ### Code ```ruby require 'mycext' time = Time.now th = Thread.new do loop do sleep 0.01 break if Time.now - time > 3 # run for 3 seconds end end th.join ``` ```c #include <pthread.h> #include <stdio.h> #include <time.h> #include "ruby.h" #include "ruby/debug.h" int called_count; void postponed_job(void *ptr) { called_count++; printf("count: %d\n", called_count); } _Noreturn void * pthread_main(void *_) { while (1) { rb_postponed_job_register_one(0, postponed_job, NULL); // Sleep for 100 ms struct timespec ts; ts.tv_sec = 0; ts.tv_nsec = 100 * 1000 * 1000; nanosleep(&ts, NULL); } } RUBY_FUNC_EXPORTED void Init_mycext(void) { called_count = 0; pthread_t pthread; pthread_create(&pthread, NULL, pthread_main, NULL); } ``` -- https://bugs.ruby-lang.org/

Issue #20197 has been updated by byroot (Jean Boussier). Backport changed from 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN to 3.0: DONTNEED, 3.1: DONTNEED, 3.2: DONTNEED, 3.3: REQUIRED @kjtsanaktsidis I'm assuming this should be backported to 3.3, but if I'm wrong you can change it to `DONTNEED` ---------------------------------------- Bug #20197: Postponed job invocations are significantly reduced in Ruby 3.3 https://bugs.ruby-lang.org/issues/20197#change-106464 * Author: osyoyu (Daisuke Aritomo) * Status: Closed * Priority: Normal * Assignee: kjtsanaktsidis (KJ Tsanaktsidis) * ruby -v: ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-linux] * Backport: 3.0: DONTNEED, 3.1: DONTNEED, 3.2: DONTNEED, 3.3: REQUIRED ---------------------------------------- The number of postponed job invocations has been significantly reduced in Ruby 3.3. While my understanding is that postponed jobs provide no guarantee of how soon registered callbacks will fire, I believe the current rate is too low for practical usage, especially for profilers such as StackProf. A git bisect led me to https://github.com/ruby/ruby/commit/1f0304218cf00e05a4a126196676ba221ebf91f6 which obviously seems to be related, but I'm not sure why. ## Repro ### Expected The job fires (nearly) immediately after being triggered. In the following example, the job is triggered every 100 ms. ``` % ruby bin/test.rb # runs for 3 seconds count: 1 count: 2 (snip) count: 29 ``` ### Actual The job gets fired only once. ``` % ruby bin/test.rb count: 1 ``` ### Code ```ruby require 'mycext' time = Time.now th = Thread.new do loop do sleep 0.01 break if Time.now - time > 3 # run for 3 seconds end end th.join ``` ```c #include <pthread.h> #include <stdio.h> #include <time.h> #include "ruby.h" #include "ruby/debug.h" int called_count; void postponed_job(void *ptr) { called_count++; printf("count: %d\n", called_count); } _Noreturn void * pthread_main(void *_) { while (1) { rb_postponed_job_register_one(0, postponed_job, NULL); // Sleep for 100 ms struct timespec ts; ts.tv_sec = 0; ts.tv_nsec = 100 * 1000 * 1000; nanosleep(&ts, NULL); } } RUBY_FUNC_EXPORTED void Init_mycext(void) { called_count = 0; pthread_t pthread; pthread_create(&pthread, NULL, pthread_main, NULL); } ``` -- https://bugs.ruby-lang.org/

Issue #20197 has been updated by naruse (Yui NARUSE). Backport changed from 3.0: DONTNEED, 3.1: DONTNEED, 3.2: DONTNEED, 3.3: REQUIRED to 3.0: DONTNEED, 3.1: DONTNEED, 3.2: DONTNEED, 3.3: DONE ruby_3_3 0793cbbfde261f4fc9bf7045594d62a21e391811 merged revision(s) ef276858d9295208add48e27208c69184dc50472. ---------------------------------------- Bug #20197: Postponed job invocations are significantly reduced in Ruby 3.3 https://bugs.ruby-lang.org/issues/20197#change-107351 * Author: osyoyu (Daisuke Aritomo) * Status: Closed * Assignee: kjtsanaktsidis (KJ Tsanaktsidis) * ruby -v: ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-linux] * Backport: 3.0: DONTNEED, 3.1: DONTNEED, 3.2: DONTNEED, 3.3: DONE ---------------------------------------- The number of postponed job invocations has been significantly reduced in Ruby 3.3. While my understanding is that postponed jobs provide no guarantee of how soon registered callbacks will fire, I believe the current rate is too low for practical usage, especially for profilers such as StackProf. A git bisect led me to https://github.com/ruby/ruby/commit/1f0304218cf00e05a4a126196676ba221ebf91f6 which obviously seems to be related, but I'm not sure why. ## Repro ### Expected The job fires (nearly) immediately after being triggered. In the following example, the job is triggered every 100 ms. ``` % ruby bin/test.rb # runs for 3 seconds count: 1 count: 2 (snip) count: 29 ``` ### Actual The job gets fired only once. ``` % ruby bin/test.rb count: 1 ``` ### Code ```ruby require 'mycext' time = Time.now th = Thread.new do loop do sleep 0.01 break if Time.now - time > 3 # run for 3 seconds end end th.join ``` ```c #include <pthread.h> #include <stdio.h> #include <time.h> #include "ruby.h" #include "ruby/debug.h" int called_count; void postponed_job(void *ptr) { called_count++; printf("count: %d\n", called_count); } _Noreturn void * pthread_main(void *_) { while (1) { rb_postponed_job_register_one(0, postponed_job, NULL); // Sleep for 100 ms struct timespec ts; ts.tv_sec = 0; ts.tv_nsec = 100 * 1000 * 1000; nanosleep(&ts, NULL); } } RUBY_FUNC_EXPORTED void Init_mycext(void) { called_count = 0; pthread_t pthread; pthread_create(&pthread, NULL, pthread_main, NULL); } ``` -- https://bugs.ruby-lang.org/