January 2023 - ruby-core - ml.ruby-lang.org

[ruby-core:112153] [Ruby master Bug#17664] Behavior of sockets changed in Ruby 3.0 to non-blocking
by ioquatix (Samuel Williams) 31 Jan '23

31 Jan '23

Issue #17664 has been updated by ioquatix (Samuel Williams). Regarding `IO.select`, my advice is to prefer `wait_readable` or `wait_writable` if possible. ---------------------------------------- Bug #17664: Behavior of sockets changed in Ruby 3.0 to non-blocking https://bugs.ruby-lang.org/issues/17664#change-101590 * Author: ciconia (Sharon Rosner) * Status: Assigned * Priority: Normal * Assignee: ioquatix (Samuel Williams) * ruby -v: 3.0.0 * Backport: 2.5: UNKNOWN, 2.6: UNKNOWN, 2.7: UNKNOWN, 3.0: UNKNOWN ---------------------------------------- I'm not sure this is a bug, but apparently a change was introduced in Ruby 3.0 that makes sockets non-blocking by default. This change was apparently introduced as part of the work on the [FiberScheduler interface](https://github.com/ruby/ruby/blame/78f188524f551c97b1a7a44ae1351…. This change of behaviour is not discussed in the Ruby 3.0.0 release notes. This change complicates the implementation of an io_uring-based fiber scheduler, since io_uring SQE's on fd's with `O_NONBLOCK` can return `EAGAIN` just like normal syscalls. Using io_uring with non-blocking fd's defeats the whole purpose of using io_uring in the first place. A workaround I have put in place in the Polyphony [io_uring backend](https://github.com/digital-fabric/polyphony/blob/d3c9cf3ddc1f41438… is to make sure `O_NONBLOCK` is not set before attempting I/O operations on any fd. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:111890] [Ruby master Bug#19352] Feature #17391 was not such a good idea because now Ruby 3.2 can not install Rails v5 or v6 if they use webpacker.
by Milella＠Hotmail.com (Scott Milella) 31 Jan '23

31 Jan '23

Issue #19352 has been reported by Milella(a)Hotmail.com (Scott Milella). ---------------------------------------- Bug #19352: Feature #17391 was not such a good idea because now Ruby 3.2 can not install Rails v5 or v6 if they use webpacker. https://bugs.ruby-lang.org/issues/19352 * Author: Milella(a)Hotmail.com (Scott Milella) * Status: Open * Priority: Normal * ruby -v: 3.2 * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Hello, I tried logging a GitHub issue to Rails to make them aware of this issue but they said they are no longer bugfixing Rails v6.1 anymore. So since this IS a Ruby issue at its core this might be the best place for this. You removed 2 methods from Ruby 3.2 namely: Dir.exists? [Feature #17391] File.exists? [Feature #17391] The reason cited was that it was deprecated since v2 and MAYBE it might be a good idea to remove it in v3. I disagree. It will alienate Rails v5 and v6 from at the very least making a new rails app because when the webpacker install is called it uses one or both of those methods. take a look at this log: Here is what I did: ``` Windows 10/11 Installed prerequisites: NodeJS, Yarn, Git, SQLite3 etc. Install Ruby on Windows using RubyInstaller v3.2.0 filename: "rubyinstaller-devkit-3.2.0-1-x64.exe" Installed newest rails via gem install rails (installed v7.0.4.1 and it successfully installed) Install rails via gem install rails -v 6.1.7 (successfully installed) rails _6.1.7_ new testapp Here is the install log around the failure point: Bundle complete! 15 Gemfile dependencies, 78 gems now installed. Use bundle info [gemname] to see where a bundled gem is installed. run bundle binstubs bundler rails webpacker:install create config/webpacker.yml Copying webpack core config create config/webpack create config/webpack/development.js create config/webpack/environment.js create config/webpack/production.js create config/webpack/test.js Copying postcss.config.js to app root directory create postcss.config.js Copying babel.config.js to app root directory create babel.config.js Copying .browserslistrc to app root directory create .browserslistrc rails aborted! NoMethodError: undefined method `exists?' for Dir:Class Tasks: TOP => app:template (See full trace by running task with --trace) ``` Is there any way we can reverse the removal of those 2 methods? Otherwise you are again alienating Rails v5 (with webpacker) and v6 which a L O T of Rails apps runs on. Why would you want to release a new version of Ruby that PREVENTS backwards compatibility. It does work properly on Ruby 3.1, but if you try Ruby 3.2 you will get the method error on exists?. Thank You, Scott -- https://bugs.ruby-lang.org/

7 15

[ruby-core:112151] [Ruby master Bug#17664] Behavior of sockets changed in Ruby 3.0 to non-blocking
by eviljoel (evil joel) 31 Jan '23

31 Jan '23

Issue #17664 has been updated by eviljoel (evil joel). @ioquatix, thank you for your reply. The sockets are created in Ruby. I'm interacting with OpenSSL via FFI, so that part is also largely in Ruby. Since I commented yesterday, I fixed the code to work with Ruby 3. I changed all relevant calls to OpenSSL to behave in a non-blocking manner leveraging IO.select. It really wasn't as difficult as I expected. Thanks again. ---------------------------------------- Bug #17664: Behavior of sockets changed in Ruby 3.0 to non-blocking https://bugs.ruby-lang.org/issues/17664#change-101588 * Author: ciconia (Sharon Rosner) * Status: Assigned * Priority: Normal * Assignee: ioquatix (Samuel Williams) * ruby -v: 3.0.0 * Backport: 2.5: UNKNOWN, 2.6: UNKNOWN, 2.7: UNKNOWN, 3.0: UNKNOWN ---------------------------------------- I'm not sure this is a bug, but apparently a change was introduced in Ruby 3.0 that makes sockets non-blocking by default. This change was apparently introduced as part of the work on the [FiberScheduler interface](https://github.com/ruby/ruby/blame/78f188524f551c97b1a7a44ae1351…. This change of behaviour is not discussed in the Ruby 3.0.0 release notes. This change complicates the implementation of an io_uring-based fiber scheduler, since io_uring SQE's on fd's with `O_NONBLOCK` can return `EAGAIN` just like normal syscalls. Using io_uring with non-blocking fd's defeats the whole purpose of using io_uring in the first place. A workaround I have put in place in the Polyphony [io_uring backend](https://github.com/digital-fabric/polyphony/blob/d3c9cf3ddc1f41438… is to make sure `O_NONBLOCK` is not set before attempting I/O operations on any fd. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:112048] [Ruby master Bug#19379] Regex: "end pattern with unmatched parenthesis" with Ruby 3.2 and interpolation
by renchap (Renaud Chaput) 31 Jan '23

31 Jan '23

Issue #19379 has been reported by renchap (Renaud Chaput). ---------------------------------------- Bug #19379: Regex: "end pattern with unmatched parenthesis" with Ruby 3.2 and interpolation https://bugs.ruby-lang.org/issues/19379 * Author: renchap (Renaud Chaput) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.0 (2022-12-25 revision a528908271) [arm64-darwin22] * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Sample code: ``` r2 = %r{#c-\w+/comment/[\w-]+} %r{https?://[^/]+#{r2}}x ``` This works with Ruby 3.1: ``` irb(main):001:0> r2 = %r{#c-\w+/comment/[\w-]+} irb(main):002:0> %r{https?://[^/]+#{r2}}x => /https?:\/\/[^\/]+(?-mix:#c-\w+\/comment\/[\w-]+)/x ``` But fails with Ruby 3.2.0: ``` irb(main):022:0> r2 = %r{#c-\w+/comment/[\w-]+} irb(main):023:0> %r{https?://[^/]+#{r2}}x (irb):23:in `<main>': end pattern with unmatched parenthesis: /https?:\/\/[^\/]+(?-mix:#c-\w+\/comment\/[\w-]+)/x (RegexpError) ``` But if I dont use interpolation, it works correctly: ``` irb(main):001:0> %r{https?://[^/]+#c-\w+/comment/[\w-]+}x => /https?:\/\/[^\/]+#c-\w+\/comment\/[\w-]+/x ``` -- https://bugs.ruby-lang.org/

5 7

[ruby-core:112114] [Ruby master Bug#19390] Time.new: Assertion Failed: bignum.c:4187:rb_int_parse_cstr:len != 0
by byroot (Jean Boussier) 31 Jan '23

31 Jan '23

Issue #19390 has been reported by byroot (Jean Boussier). ---------------------------------------- Bug #19390: Time.new: Assertion Failed: bignum.c:4187:rb_int_parse_cstr:len != 0 https://bugs.ruby-lang.org/issues/19390 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.0 (2023-01-25 revision a22eca8231), ruby 3.3.0dev (2023-01-30T08:29:31Z master 1c7bf378a1) * Backport: 2.7: DONTNEED, 3.0: DONTNEED, 3.1: DONTNEED, 3.2: REQUIRED ---------------------------------------- Can be reproduced with the following snippet: ```ruby Time.new("2020-10-28T16:48:07.000Z") ``` I suspect but haven't confirmed that this bug was introduced in the fix for `[Bug #19296]` cc @nobu I have a patch for it, I'll open a PR in a couple minutes. -- https://bugs.ruby-lang.org/

2 2

[ruby-core:112127] [Ruby master Bug#17664] Behavior of sockets changed in Ruby 3.0 to non-blocking
by ioquatix (Samuel Williams) 31 Jan '23

31 Jan '23

Issue #17664 has been updated by ioquatix (Samuel Williams). @eviljoel thanks for your report. My initial thoughts are: - If you are using Ruby to construct sockets, you should probably use the Ruby methods to accept connections, etc. They will correctly deal with this. At worst, you might be blocking without releasing the GVL. Better to use `rb_io_wait`. - I don't think we will roll back this change but it's not impossible - in theory it should not be externally visible. The reason for non-blocking IO is nuanced and complex, and we might eventually be able to achieve what we want without making all file descriptors non-blocking. - You can set a socket back to blocking if you don't intend for Ruby code to use it directly, but make sure you are handling releasing the GVL. ---------------------------------------- Bug #17664: Behavior of sockets changed in Ruby 3.0 to non-blocking https://bugs.ruby-lang.org/issues/17664#change-101563 * Author: ciconia (Sharon Rosner) * Status: Assigned * Priority: Normal * Assignee: ioquatix (Samuel Williams) * ruby -v: 3.0.0 * Backport: 2.5: UNKNOWN, 2.6: UNKNOWN, 2.7: UNKNOWN, 3.0: UNKNOWN ---------------------------------------- I'm not sure this is a bug, but apparently a change was introduced in Ruby 3.0 that makes sockets non-blocking by default. This change was apparently introduced as part of the work on the [FiberScheduler interface](https://github.com/ruby/ruby/blame/78f188524f551c97b1a7a44ae1351…. This change of behaviour is not discussed in the Ruby 3.0.0 release notes. This change complicates the implementation of an io_uring-based fiber scheduler, since io_uring SQE's on fd's with `O_NONBLOCK` can return `EAGAIN` just like normal syscalls. Using io_uring with non-blocking fd's defeats the whole purpose of using io_uring in the first place. A workaround I have put in place in the Polyphony [io_uring backend](https://github.com/digital-fabric/polyphony/blob/d3c9cf3ddc1f41438… is to make sure `O_NONBLOCK` is not set before attempting I/O operations on any fd. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:112126] [Ruby master Bug#17664] Behavior of sockets changed in Ruby 3.0 to non-blocking
by eviljoel (evil joel) 30 Jan '23

30 Jan '23

Issue #17664 has been updated by eviljoel (evil joel). @ioquatix, I maintain a proprietary, custom OpenSSL C extension as part of my company's security scanner. This change broke the unit tests for our C extension. (This change probably also delayed our upgrade to Ruby 3 from Ruby 2.7 due to the complexity of debugging this issue.) OpenSSL's SSL_accept function inherits the blocking behavior of the socket file descriptor. Since our C extension unit tests relied on SSL_accept being blocking, upgrading to Ruby 3+ broke our unit tests. (Also, it took me a while to figure out it was just the unit tests that broke and not the code being tested.) This "bug" has been around for some time now and is estalished behavior, so I don't expect this aspect of the Fiber implementation to be reverted, but I did want to raise the fact that this undocumented change did break somebody's code somewhere. Also, I wanted to ask whether it is an acceptable work around to set the socket file descriptor back to blocking (via C)? Would doing this have unforeseen consequences if I'm *not* using Fibers? (I know this isn't the best place to ask support questions but others are likely to come here with the same question.) Thank you. ---------------------------------------- Bug #17664: Behavior of sockets changed in Ruby 3.0 to non-blocking https://bugs.ruby-lang.org/issues/17664#change-101562 * Author: ciconia (Sharon Rosner) * Status: Assigned * Priority: Normal * Assignee: ioquatix (Samuel Williams) * ruby -v: 3.0.0 * Backport: 2.5: UNKNOWN, 2.6: UNKNOWN, 2.7: UNKNOWN, 3.0: UNKNOWN ---------------------------------------- I'm not sure this is a bug, but apparently a change was introduced in Ruby 3.0 that makes sockets non-blocking by default. This change was apparently introduced as part of the work on the [FiberScheduler interface](https://github.com/ruby/ruby/blame/78f188524f551c97b1a7a44ae1351…. This change of behaviour is not discussed in the Ruby 3.0.0 release notes. This change complicates the implementation of an io_uring-based fiber scheduler, since io_uring SQE's on fd's with `O_NONBLOCK` can return `EAGAIN` just like normal syscalls. Using io_uring with non-blocking fd's defeats the whole purpose of using io_uring in the first place. A workaround I have put in place in the Polyphony [io_uring backend](https://github.com/digital-fabric/polyphony/blob/d3c9cf3ddc1f41438… is to make sure `O_NONBLOCK` is not set before attempting I/O operations on any fd. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:112124] [Ruby master Feature#15778] Expose an API to pry-open the stack frames in Ruby
by st0012 (Stan Lo) 30 Jan '23

30 Jan '23

Issue #15778 has been updated by st0012 (Stan Lo). I had a chance to chat with multiple people about this proposal, so I want to share what I found here: - From chatting with @maximecb, my understanding is that such feature requires Ruby to keep all the frames on the stack in an accessible format, which will limit the optimisation YJIT could do. This is fine for development, but using it in production with YJIT "might" not be ideal. - Rails would be happy to use such feature to enrich its exception backtrace in log, which will be helpful in development. - In production, in addition to the potential issue with YJIT, more things like data scrubbing needs to be considered. ---------------------------------------- Feature #15778: Expose an API to pry-open the stack frames in Ruby https://bugs.ruby-lang.org/issues/15778#change-101560 * Author: gsamokovarov (Genadi Samokovarov) * Status: Open * Priority: Normal * Assignee: ko1 (Koichi Sasada) ---------------------------------------- Hello, I'm the maintainer of the web-console (https://github.com/rails/web-console/) gem, where one of our features is to jump between the frames in which an error occurred. To accomplish this, I currently use the Debug Inspector CRuby API. I think we should expose this functionality in Rubyland, so tools like web-console don't need to resort to C code for this. This also makes it quite harder for me to support different implementations like JRuby or TruffleRuby as everyone is having a different way to create Ruby Binding objects that represent the frames. Here the API ideas: Add `Thread::Backtrace::Location#binding` method that can create a binding for a specific caller of the current frame. We can reuse the existing `Kernel.caller_locations` method to generate the array of `Thread::Backtrace::Location` objects. We can optionally have the `Kernel.caller_locations(debug: true)` argument if we cannot generate the bindings lazily on the VM that can instruct the VM to do the slower operation. - `Thread::Backtrace::Location#binding` returns `Binding|nil`. Nil result may mean that the current location is a C frame or a JITted/optimized frame and we cannot debug it. We can also expose the DebugInspector API directly, as done in the https://github.com/banister/debug_inspector gem, but for tools like web-console, we'd need to map the bindings with the backtrace, as we cannot generate Bindings for every frame (C frames) and this needs to be done in application code, so I think the `Thread::Backtrace::Location#binding` is the better API for Ruby-land. Such API can help us eventually write most of our debuggers in Ruby as right now we don't have a way to do Post-Mortem debugging without native code or even start our debuggers without monkey-patching `Binding`. I have presented this idea in a RubyKaigi's 2019 talk called "Writing Debuggers in Plain Ruby", you can check-out the slides for more context: http://kaigi-debuggers-in-ruby.herokuapp.com. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:112115] [Ruby master Misc#19391] IO #write/#read behaviour when binmode and encoding are explicitly specified
by andrykonchin (Andrew Konchin) 30 Jan '23

30 Jan '23

Issue #19391 has been reported by andrykonchin (Andrew Konchin). ---------------------------------------- Misc #19391: IO #write/#read behaviour when binmode and encoding are explicitly specified https://bugs.ruby-lang.org/issues/19391 * Author: andrykonchin (Andrew Konchin) * Status: Open * Priority: Normal ---------------------------------------- I have a question about how `IO` operations are supposed to behave when file is opened in binary mode but encoding is specified too. Documentation of `b` option of `IO.new` method and of `IO#binmode` method is a bit not synchronised and confusing: `IO.open`: ``` "b" Binary file mode Suppresses EOL <-> CRLF conversion on Windows. And sets external encoding to ASCII-8BIT unless explicitly specified. ``` `IO#binmode`: ``` Puts ios into binary mode. Once a stream is in binary mode, it cannot be reset to nonbinary mode. newline conversion disabled encoding conversion disabled content is treated as ASCII-8BIT ``` So `unless explicitly specified` and `encoding conversion disabled` are a bit contradictory. Right now I observe that `IO.write` still encodes an input string in explicitly specified encoding (with `encoding` option) even if binary mode is set (with `binmode` option) ```ruby File.open(@filename, "w", encoding: Encoding::UTF_32LE, binmode: true) do |file| file.write("hi") end File.binread(@filename) # => "h\u0000\u0000\u0000i\u0000\u0000\u0000" ``` -- https://bugs.ruby-lang.org/

1 0

[ruby-core:112113] [Ruby master Feature#18285] NoMethodError#message uses a lot of CPU/is really expensive to call
by mame (Yusuke Endoh) 30 Jan '23

30 Jan '23

Issue #18285 has been updated by mame (Yusuke Endoh). I have updated my PR: https://github.com/ruby/ruby/pull/6950 As I was working on updating the PR, the following message change caught my attention. ``` old: undefined method `foo' for main:Object new: undefined method `foo' for #<Object:0x00007f230602cd28> ``` Because the toplevel (`main`) is an extended object, `any_to_s` rule is applied. But I think the new message is too confusing. I talked with @matz, and we decided to handle the toplevel specially: ``` new: undefined method `foo' for main ``` I found no other confusing examples during updating my PR. ---------------------------------------- Feature #18285: NoMethodError#message uses a lot of CPU/is really expensive to call https://bugs.ruby-lang.org/issues/18285#change-101550 * Author: ivoanjo (Ivo Anjo) * Status: Open * Priority: Normal ---------------------------------------- Hello there! I'm working at Datadog on the ddtrace gem -- https://github.com/DataDog/dd-trace-rb and we ran into this issue on one of our internal testing applications. I also blogged about this issue in <https://ivoanjo.me/blog/2021/11/01/nomethoderror-ruby-cost/>. ### Background While testing an application that threw a lot of `NoMethodError`s in a Rails controller (this was used for validation), we discovered that service performance was very much impacted when we were logging these exceptions. While investigating with a profiler, the performance impact was caused by calls to `NoMethodError#message`, because this Rails controller had a quite complex `#inspect` method, that was getting called every time we tried to get the `#message` from the exception. ### How to reproduce ```ruby require 'bundler/inline' gemfile do source 'https://rubygems.org' gem 'benchmark-ips' end puts RUBY_DESCRIPTION class GemInformation # ... def get_no_method_error method_does_not_exist rescue => e e end def get_runtime_error raise 'Another Error' rescue => e e end def inspect # <-- expensive method gets called when calling NoMethodError#message Gem::Specification._all.inspect end end NO_METHOD_ERROR_INSTANCE = GemInformation.new.get_no_method_error RUNTIME_ERROR_INSTANCE = GemInformation.new.get_runtime_error Benchmark.ips do |x| x.config(:time => 5, :warmup => 2) x.report("no method error message cost") { NO_METHOD_ERROR_INSTANCE.message } x.report("runtime error message cost") { RUNTIME_ERROR_INSTANCE.message } x.compare! end ``` ### Expectation and result Getting the `#message` from a `NoMethodError` should be no costly than getting it from any other exception. In reality: ``` ruby 3.0.2p107 (2021-07-07 revision 0db68f0233) [x86_64-linux] no method error message cost 115.390 (± 1.7%) i/s - 580.000 in 5.027822s runtime error message cost 6.938M (± 0.5%) i/s - 35.334M in 5.092617s Comparison: runtime error message cost: 6938381.6 i/s no method error message cost: 115.4 i/s - 60130.02x (± 0.00) slower ``` ### Suggested solutions 1. Do not call `#inspect` on the object on which the method was not found (see <https://github.com/ruby/ruby/blob/e0915ba67964d843832148aeca29a1f8244ca7b1/…>) 2. Cache result of calling `#message` after the first call. Ideally this should be done together with suggestion 1. -- https://bugs.ruby-lang.org/

1 0