August 2023 - ruby-core - ml.ruby-lang.org

[ruby-core:112326] [Ruby master Feature#19430] Contribution wanted: DNS lookup by c-ares library
by mame (Yusuke Endoh) 19 Oct '23

19 Oct '23

Issue #19430 has been reported by mame (Yusuke Endoh). ---------------------------------------- Feature #19430: Contribution wanted: DNS lookup by c-ares library https://bugs.ruby-lang.org/issues/19430 * Author: mame (Yusuke Endoh) * Status: Open * Priority: Normal ---------------------------------------- ## Problem At the present time, Ruby uses `getaddrinfo(3)` to resolve names. Because this function is synchronous, we cannot interrupt the thread performing name resolution until the DNS server returns a response. We can see this behavior by setting blackhole.webpagetest.org (72.66.115.13) as a DNS server, which swallows all packets, and resolving any name: ``` # cat /etc/resolv.conf nameserver 72.66.115.13 # ./local/bin/ruby -rsocket -e 'Addrinfo.getaddrinfo("www.ruby-lang.org", 80)' ^C^C^C^C ``` As we see, Ctrl+C does not stop ruby. The current workaround that users can take is to do name resolution in a Ruby thread. ```ruby Thread.new { Addrinfo.getaddrinfo("www.ruby-lang.org", 80) }.value ``` The thread that calls this code is interruptible. (Note that the newly created thread itself will be stuck until the DNS lookup exceeds the time out.) ## Proposal We can solve this problem by using c-ares, which is an asynchronous name resolver, as a backend of `Addrinfo.getaddrinfo`, etc. (@sorah told me about this library, thanks!) https://c-ares.org/ I have created a PoC patch. https://github.com/mame/ruby/commit/547806146993bbc25984011d423dcc0f913b211c By applying this patch, we can interrupt `Addrinfo.getaddrinfo` by Ctrl+C. ``` # cat /etc/resolv.conf nameserver 72.66.115.13 # ./local/bin/ruby -rsocket -e 'Addrinfo.getaddrinfo("www.ruby-lang.org", 80)' ^C-e:1:in `getaddrinfo': Interrupt from -e:1:in `<main>' ``` ## Discussion ### About c-ares According to the site of c-ares, some major tools including libcurl, Wireshark, and Apache Arrow are already using c-ares. In the language interpreter, node.js seems to be using c-ares. I am honestly not sure about the compatibility of c-ares with `getaddrinfo(3)`. I guess there is no major incompatibility because I have not experienced any name resolution problem of curl. @akr (who is the author and maintainer of Ruby's socket library) suggested to check if OS-specific name resolution, e.g., WINS on Windows, NIS on Solaris, etc., is supported. He also said that it may be acceptable even if they are not supported. Whether to bundle c-ares source code with ruby would require further discussion. If this proposal is accepted, then c-ares will become a de facto essential dependency for practical use, like gmp, in my opinion. Incidentally, node.js bundles c-ares: https://github.com/nodejs/node/tree/main/deps/cares ### Alternative approaches Recent glibc provides `getaddrinfo_a(3)` which performs asynchronous name resolution. However, this function has a fatal problem of being incompatible with `fork(2)`, which is heavily used in the Ruby ecosystem. In fact, the attempt to use `getaddrinfo_a(3)` (#17134) has been revert because it fails rails tests. (#17220) Another alternative is to have a worker pthread inside Ruby that calls getaddrinfo(3). Instead of calling getaddrinfo(3) directly, `Addrinfo.getaddrinfo` would ask the worker to resolve a name and wait for a response. This method should be able to implement cancellation. (Simply put, this means reimplementation of getaddrinfo_a(3) on our own, taking into account of `fork(2).) This has the advantages: not adding dependencies on external libraries and not having compatibility issues with `getaddrinfo(3)`. However, it is considerably more difficult to implement and maintain. An internal pthread may have a non-trivial impact on the execution efficiency and memory usage. Also, we may need to implement a mechanism to dynamically change the number of workers depending on the load. It would be ideal if we could try and evaluate both approaches. But my current impression is that using c-ares is the quickest and best compromise. ## Contribution wanted I have made it up to the PoC, but don't have much time to complete this. @naruse suggested me to create a ticket asking for contributions. Is anyone interested in this? * This patch changes `rsock_getaddrinfo` to accept a timeout argument. There are several places where Qnil is passed as a timeout (where I add `// TODO` in the PoC). We need to consider what timeout we should pass. * This cares only `getaddrinfo`, but we also need to care `getnameinfo` (and something else if any). There may be some issues I'm not aware of. * I have not yet tested this PoC seriously. It would be great if we could evaluate it with some real apps. Also, it would be great to hear from someone who knows more about c-ares. -- https://bugs.ruby-lang.org/

6 15

[ruby-core:114250] [Ruby master Bug#19778] mkmf.rb pkg_config() interaction with RbConfig::CONFIG["cflags"]
by rhenium (Kazuki Yamaguchi) 17 Oct '23

17 Oct '23

Issue #19778 has been reported by rhenium (Kazuki Yamaguchi). ---------------------------------------- Bug #19778: mkmf.rb pkg_config() interaction with RbConfig::CONFIG["cflags"] https://bugs.ruby-lang.org/issues/19778 * Author: rhenium (Kazuki Yamaguchi) * Status: Open * Priority: Normal * ruby -v: ruby 3.3.0dev (2023-07-21T09:38:29Z master 22f9735587) [x86_64-linux] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- (This was first reported at https://github.com/ruby/openssl/issues/650: The extension's build breaks in a strange way if `RbConfig::CONFIG["*flags"]` contains the path of an OpenSSL installation but `pkg-config` returns the path of a different OpenSSL installation) Commit commit:097c3e9cbbf23718371f08c24b2d2297b039f63f ("mkmf.rb: -I flags to $INCFLAGS", Ruby 2.2) changed how mkmf's `pkg_config()` handles the result of the `pkg-config` command. It now stores the `-I` flags in $INCFLAGS and others in $CFLAGS. mkmf generates a Makefile that compiles source files with: $(CC) $(INCFLAGS) $(CPPFLAGS) $(CFLAGS) and link the final library with: $(LDSHARED) -o $@ $(OBJS) $(LIBPATH) $(DLDFLAGS) $(LOCAL_LIBS) $(LIBS) This "new" behavior of `pkg_config()` is problematic when `RbConfig::CONFIG["{C,CPP}FLAGS"]` also provide `-I` flags and `RbConfig::CONFIG["LDFLAGS"]` provides the matching `-L` flags -- for example, if Ruby is compiled with `./configure --with-opt-dir=<dir>`. This would end up with compiling sources with [...] -I<from pkg-config> -I<from RbConfig> [...] and then linking with [...] -L<from RbConfig> -L<from pkg-config> [...] This doesn't seem right. I don't know which should come earlier, but the order should be consistent. The commit in question clearly describes the change in the commit message, but it doesn't have a linked issue. What is it intended for? Also, what is $INCFLAGS? On the other hand, `dir_config()` would prepend `-I` flags to $CPPFLAGS and `-L` flags to $LIBPATH, so it doesn't have issues with flags from `RbConfig`, albeit in a differently way. -- https://bugs.ruby-lang.org/

5 4

[ruby-core:114399] [Ruby master Feature#19839] Need a method to check if two ranges overlap
by shouichi (Shouichi KAMIYA) 13 Oct '23

13 Oct '23

Issue #19839 has been reported by shouichi (Shouichi KAMIYA). ---------------------------------------- Feature #19839: Need a method to check if two ranges overlap https://bugs.ruby-lang.org/issues/19839 * Author: shouichi (Shouichi KAMIYA) * Status: Open * Priority: Normal ---------------------------------------- It would be convenient to have a method that checks if two ranges overlap. For example, ``` (0..10).overlap?(5..15) #=> true (0..10).overlap?(20..30) #=> false ``` -- https://bugs.ruby-lang.org/

12 26

[ruby-core:114574] [Ruby master Bug#19855] Array#bsearch gets wrong result when its block returns a non-integer numeric value.
by kyanagi (Kouhei Yanagita) 10 Oct '23

10 Oct '23

Issue #19855 has been reported by kyanagi (Kouhei Yanagita). ---------------------------------------- Bug #19855: Array#bsearch gets wrong result when its block returns a non-integer numeric value. https://bugs.ruby-lang.org/issues/19855 * Author: kyanagi (Kouhei Yanagita) * Status: Open * Priority: Normal * ruby -v: ruby 3.3.0dev (2023-08-29T04:52:18Z master c0e913ae88) [arm64-darwin22] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Array#bsearch gets wrong result when its block returns a non-integer numeric value. ``` % ruby -e 'p [0, 4, 7, 10, 12].bsearch { |x| (4 - x) }' 4 % ruby -e 'p [0, 4, 7, 10, 12].bsearch { |x| (4 - x).to_r }' nil ``` fix: https://github.com/ruby/ruby/pull/8314 -- https://bugs.ruby-lang.org/

3 3

[ruby-core:114475] [Ruby master Bug#19845] `Fiber[key] = value` fails if key is not interned.
by ioquatix (Samuel Williams) 30 Sep '23

30 Sep '23

Issue #19845 has been reported by ioquatix (Samuel Williams). ---------------------------------------- Bug #19845: `Fiber[key] = value` fails if key is not interned. https://bugs.ruby-lang.org/issues/19845 * Author: ioquatix (Samuel Williams) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- See https://github.com/ruby/ruby/pull/8273 for the fix. -- https://bugs.ruby-lang.org/

2 2

[ruby-core:114181] [Ruby master Bug#19767] [Not really a bug, but more a not ideal notification] "historical binary regexp match" when using the "n" modifier in a ruby regex
by rubyFeedback (robert heiler) 29 Sep '23

29 Sep '23

Issue #19767 has been reported by rubyFeedback (robert heiler). ---------------------------------------- Bug #19767: [Not really a bug, but more a not ideal notification] "historical binary regexp match" when using the "n" modifier in a ruby regex https://bugs.ruby-lang.org/issues/19767 * Author: rubyFeedback (robert heiler) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- To get my knowledge about ruby regexes up-to-date I have been going through this tutorial/book here at: https://learnbyexample.github.io/Ruby_Regexp/unicode.html One example they provide is this, with some odd characters: 'fox:αλεπού'.scan(/\w+/n) This will match the found word ("fox"), but it also reports the following warning: warning: historical binary regexp match /.../n against UTF-8 string Now: this may be obvious to others, but to me personally I am not sure what a "historical" binary regexp match actually is. I assume it may have meant that this was more used in the past, and may be discouraged now? Or is something else meant? What does "historical" mean in this context? I may not be the only one who does not fully understand the term historical. Most of ruby's warnings are fairly easy to understand, but this one seems odd. Right now I do not know whether we can use the "n" modifier in a regex - not that I really have a good use case for it (I am using UTF-8 these days, so I don't seem to need ASCII-8BIT anyway), but perhaps the warning could be changed a little. I have no good alternative suggestion how it can be changed, largely because I do not know what it actually means, e. g. what is "historical" about it (but, even then, I'd actually recommend against using the word "historical" because I don't understand what it means; deprecated is easy to understand, historical does not tell me anything). Perhaps it could be expressed somewhat differently and we could get rid of the word "historical" there? Either way, it's a tiny issue so I was not even sure whether to report it. But, from the point of view of other warnings, I believe the term "historical" does not tell the user enough about what the issue is here. (irb):1: warning: historical binary regexp match /.../n against UTF-8 string => ["fox"] -- https://bugs.ruby-lang.org/

3 2

[ruby-core:114568] [Ruby master Bug#19853] Segfault when calling `system` from a non-blocking fiber
by rmosolgo (Robert Mosolgo) 24 Sep '23

24 Sep '23

Issue #19853 has been reported by rmosolgo (Robert Mosolgo). ---------------------------------------- Bug #19853: Segfault when calling `system` from a non-blocking fiber https://bugs.ruby-lang.org/issues/19853 * Author: rmosolgo (Robert Mosolgo) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-darwin22] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Hi! I'm trying to improve my Ruby library to support transfer-based fibers (https://github.com/rmosolgo/graphql-ruby/pull/4322) To test parallel I/O, I'm using `system("sleep #{duration}")`, but this causes a segfault sometimes. I'd love to find a way to keep developing this feature but I'm not sure where to go next. I have attached a script which replicates the bug and I have attached the full segfault error dump. I encountered this error by running: ``` ruby fiber_bug.rb 2>out.txt ``` Does that help debug? Please let me know if I can provide anything else that'd be helpful. ---Files-------------------------------- out.txt (121 KB) fiber_bug.rb (624 Bytes) -- https://bugs.ruby-lang.org/

3 5

[ruby-core:114222] [Ruby master Misc#19772] API Naming for YARP compiler
by jemmai (Jemma Issroff) 22 Sep '23

22 Sep '23

Issue #19772 has been reported by jemmai (Jemma Issroff). ---------------------------------------- Misc #19772: API Naming for YARP compiler https://bugs.ruby-lang.org/issues/19772 * Author: jemmai (Jemma Issroff) * Status: Open * Priority: Normal ---------------------------------------- We are working on the YARP compiler, and have [the first PR ready](https://github.com/ruby/ruby/pull/8042) which introduces the YARP compile method. Our only outstanding question before merging it is about naming. How should we expose the public API for YARP's compile method? Potential suggestions: 1. YARP.compile 2. RubyVM::InstructionSequence.compile(yarp: true) 3. RubyVM::InstructionSequence.compile_yarp 4. Any of the above options, with a name other than yarp (please suggest an alternative) Regarding option 1, which would mirror `YARP.parse`, is the top level constant `YARP` acceptable? cc @matz @ko1 -- https://bugs.ruby-lang.org/

9 30

[ruby-core:114332] [Ruby master Bug#19794] Ruby 3.2.2 fails to build on macOS Sonoma betas
by jhaungs (Jim Haungs) 21 Sep '23

21 Sep '23

Issue #19794 has been reported by jhaungs (Jim Haungs). ---------------------------------------- Bug #19794: Ruby 3.2.2 fails to build on macOS Sonoma betas https://bugs.ruby-lang.org/issues/19794 * Author: jhaungs (Jim Haungs) * Status: Open * Priority: Normal * ruby -v: 3.2.2 * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- With Big Sur, Apple deprecated putting dylibs in /usr/local/lib. In Sonoma (beta 4), this directory has disappeared completely. However, ruby's configure script depends on its existence. So, virtually every ruby installer (RVM, rbenv, asdf, ruby-build, and even building from source tarball) fails. When building ruby 3.2.2 from source, the configure step outputs the irritatingly useless "something wrong with LDFLAGS" error message and fails to build. The solution was to `cd /usr/local; sudo mkdir lib` to create the missing lib directory under /usr/local. It would be nice to remove this dependency from the configure script. -- https://bugs.ruby-lang.org/

5 6

[ruby-core:113381] [Ruby master Bug#19624] Backticks - IO object leakage
by pineman 20 Sep '23

20 Sep '23

Issue #19624 has been reported by pineman (João Pinheiro). ---------------------------------------- Bug #19624: Backticks - IO object leakage https://bugs.ruby-lang.org/issues/19624 * Author: pineman (João Pinheiro) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.2 (2023-03-30 revision e51014f9c0) [arm64-darwin22] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Hi, This code works on ruby 3.0.6: ```ruby `echo` ObjectSpace.each_object(IO) do |io| if ![STDIN, STDOUT, STDERR].include?(io) io.close end end ``` but raises `IOError` on 3.2.2: ``` minimal-repro-case.rb:8:in `close': uninitialized stream (IOError) ``` I found it started failing on ruby 3.1.0 and after, on macOS and Linux. This code is useful for closing unneeded IO objects in forked processes. It looks like backticks is 'leaking' IO objects, waiting for GC, and it didn't used to in 3.1.0. In ruby 3.1.0, inside `rb_f_backquote`, `rb_gc_force_recycle` was removed in favor of `RB_GC_GUARD`. I wonder if this has something to do with the problem. Is this code incorrect since ruby 3.1.0 or is it a bug in ruby? Thanks. ---Files-------------------------------- minimal-repro-case.rb (109 Bytes) -- https://bugs.ruby-lang.org/

4 9