October 2023 - ruby-core - ml.ruby-lang.org

[ruby-core:112944] [Ruby master Feature#19541] Proposal: Generate frame unwinding info for YJIT code
by kjtsanaktsidis (KJ Tsanaktsidis) 19 Oct '23

19 Oct '23

Issue #19541 has been reported by kjtsanaktsidis (KJ Tsanaktsidis). ---------------------------------------- Feature #19541: Proposal: Generate frame unwinding info for YJIT code https://bugs.ruby-lang.org/issues/19541 * Author: kjtsanaktsidis (KJ Tsanaktsidis) * Status: Open * Priority: Normal ---------------------------------------- ## What is being propsed? Currently, Ruby crashes with yjit generated code on the stack, `rb_print_backtrace()` is unable to actually show any frames underneath the yjit code. For example, if you send SIGSEGV to a Ruby process running yjit, this is what you see: ``` /ruby/miniruby(rb_print_backtrace+0xc) [0xaaaad0276884] /ruby/vm_dump.c:785 /ruby/miniruby(rb_vm_bugreport) /ruby/vm_dump.c:1093 /ruby/miniruby(rb_bug_for_fatal_signal+0xd0) [0xaaaad0075580] /ruby/error.c:813 /ruby/miniruby(sigsegv+0x5c) [0xaaaad01bedac] /ruby/signal.c:919 linux-vdso.so.1(__kernel_rt_sigreturn+0x0) [0xffff91a3e8bc] /ruby/miniruby(map<(usize, yjit::backend::ir::Insn), (usize, yjit::backend::ir::Insn), yjit::backend::ir::{impl#17}::next_mapped::{closure_env#0}>+0x8c) [0xaaaad03b8b00] /rustc/897e37553bba8b42751c67658967889d11ecd120/library/core/src/option.rs:929 /ruby/miniruby(next_mapped+0x3c) [0xaaaad0291dc0] src/backend/ir.rs:1225 /ruby/miniruby(arm64_split+0x114) [0xaaaad0287744] src/backend/arm64/mod.rs:359 /ruby/miniruby(compile_with_regs+0x80) [0xaaaad028bf84] src/backend/arm64/mod.rs:1106 /ruby/miniruby(compile+0xc4) [0xaaaad0291ae0] src/backend/ir.rs:1158 /ruby/miniruby(gen_single_block+0xe44) [0xaaaad02b1f88] src/codegen.rs:854 /ruby/miniruby(gen_block_series_body+0x9c) [0xaaaad03b0250] src/core.rs:1698 /ruby/miniruby(gen_block_series+0x50) [0xaaaad03b0100] src/core.rs:1676 /ruby/miniruby(branch_stub_hit_body+0x80c) [0xaaaad03b1f68] src/core.rs:2021 /ruby/miniruby({closure#0}+0x28) [0xaaaad02eb86c] src/core.rs:1924 /ruby/miniruby(do_call<yjit::core::branch_stub_hit::{closure_env#0}, *const u8>+0x98) [0xaaaad035ba3c] /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/panicking.rs:492 [0xaaaad035c9b4] ``` (n.b. - I compiled Ruby with `-fasynchronous-unwind-tables –rdynamic –g` in cflags to make sure gcc generates appropriate unwind info & keeps the symbol tables). Likewise, if you attach gdb to a Ruby process with yjit enabled, gdb can't show thread backtraces through yjit-generated code either. My proposal is that YJIT generate sufficient unwinding and debug information on all platforms to allow both `rb_print_backtrace()` and the platform's debugger (gdb/lldb/WinDbg) to show: * Full stack traces all the way back to `main`. That is, it should be possible to see frames _underneath_ `[0xaaaad035c9b4]` from the backtrace above. * Names for the dynamically generated yjit blocks (e.g. instead of `[0xaaaad035c9b4]`, we should see something like `yjit$$name_of_ruby_method`, where `name_of_ruby_method` is the `label` for the iseq this is JIT'd code for). ## Motivation I have a few motivations for wanting this. Firstly, I feel this functionality is independently useful. When Ruby crashes, the more information we can get, the more likely we are to find the root cause. Likewise, the same principle applies to debugging with gdb - you can get a fuller understanding of what the process is doing if you see the whole stack. I have often found attaching gdb to the Ruby interpreter helps in understanding problems in Ruby code or C extensions and is something I do relatively frequently; yjit breaking that will definitely be inconvenient for me! ## Implementation I have a draft implementation here on how I'd implement this: . It's currently missing tests & platform support (it only works on Linux aarch64). Also, it implements unwind info generation, so unwinding can work _through_ yjit code, but it does not currently emit symbols to give _names_ to those yjit frames. My PR contains a document which explains how the Linux interfaces for registering unwind info for JIT'd code work, so I won't duplicate that information here. The biggest implementation question I had is around the use of Rust crates. Currently, I prototyped my implementation using the gimli & object crates, for generating DWARF info and ELF binaries. However, the yjit build does purposefully does not use cargo & external crates for release builds. There are a few different ways we could go here: * Don't use the gimli & object crates; instead, re-implement all debug info & object file generation code in yjit. * Don't use the crates; instead, link againt C libraries to provide this functionality & call them from Rust (perhaps some combination of libelf, libdw, libbfd, or llvm might do what we need) * Use cargo after all for the release build & download the crates at build-time * Use cargo for the release build, but vendor everything, so the build doesn't need to download anything * Only make unwind info generation available in dev mode where cargo is used, and so mark the gimli/object dependencies as optional in Cargo.toml. We'd need to decide on one of these approaches for this proposal to work. I don't really have a strong sense of the pros/cons of each. (Side note - my PR actually depends on a _fork_ of gimli - I've been discussing adding the needed interfaces upstream here: https://github.com/gimli-rs/gimli/issues/648) ## Benchmarks I ran the yit-bench suite on my branch and compared it to Ruby master: * My branch: https://gist.github.com/KJTsanaktsidis/5741a9f64e5cd75cdf5fedd846091a4f * Ruby master: https://gist.github.com/KJTsanaktsidis/592d3ebcf98f6745dfa3efbd30a25acf This is a (very simple) comparison: ``` -------------- ------------ ------------ --------------- bench yjit (ms) branch (ms) branch/yjit (%) activerecord 97.5 98.5 101.03% hexapdf 2415.3 2458.2 101.78% liquid-c 61.9 63.1 101.94% liquid-render 135.3 135.0 99.78% mail 104.6 105.5 100.86% psych-load 1887.1 1922.0 101.85% railsbench 1544.4 1556.0 100.75% ruby-lsp 88.4 89.5 101.24% sequel 147.5 151.1 102.44% binarytrees 303 305.6 100.86% chunky_png 1075.8 1079.4 100.33% erubi 392.9 392.3 99.85% erubi_rails 14.7 14.7 100.00% etanni 792.3 791.4 99.89% fannkuchredux 3815.9 3813.6 99.94% lee 1030.2 1039.2 100.87% nbody 49.2 49.3 100.20% optcarrot 4142 4143.3 100.03% ruby-json 2860.7 2874.0 100.46% rubykon 7906.6 7904.2 99.97% 30k_ifelse 348.7 345.4 99.05% 30k_methods 828.6 831.8 100.39% cfunc_itself 28.8 28.9 100.35% fib 34.4 34.5 100.29% getivar 115.5 109.7 94.98% keyword_args 37.7 38.0 100.80% respond_to 26 26.1 100.38% setivar 33.8 33.5 99.11% setivar_object 208.7 194.3 93.10% str_concat 52.6 52.2 99.24% throw 23.8 24.1 101.26% -------------- ------------ ------------ --------------- ``` It seems like the performance impact of generating and registering the debug info is marginal. -- https://bugs.ruby-lang.org/

7 12

[ruby-core:112326] [Ruby master Feature#19430] Contribution wanted: DNS lookup by c-ares library
by mame (Yusuke Endoh) 19 Oct '23

19 Oct '23

Issue #19430 has been reported by mame (Yusuke Endoh). ---------------------------------------- Feature #19430: Contribution wanted: DNS lookup by c-ares library https://bugs.ruby-lang.org/issues/19430 * Author: mame (Yusuke Endoh) * Status: Open * Priority: Normal ---------------------------------------- ## Problem At the present time, Ruby uses `getaddrinfo(3)` to resolve names. Because this function is synchronous, we cannot interrupt the thread performing name resolution until the DNS server returns a response. We can see this behavior by setting blackhole.webpagetest.org (72.66.115.13) as a DNS server, which swallows all packets, and resolving any name: ``` # cat /etc/resolv.conf nameserver 72.66.115.13 # ./local/bin/ruby -rsocket -e 'Addrinfo.getaddrinfo("www.ruby-lang.org", 80)' ^C^C^C^C ``` As we see, Ctrl+C does not stop ruby. The current workaround that users can take is to do name resolution in a Ruby thread. ```ruby Thread.new { Addrinfo.getaddrinfo("www.ruby-lang.org", 80) }.value ``` The thread that calls this code is interruptible. (Note that the newly created thread itself will be stuck until the DNS lookup exceeds the time out.) ## Proposal We can solve this problem by using c-ares, which is an asynchronous name resolver, as a backend of `Addrinfo.getaddrinfo`, etc. (@sorah told me about this library, thanks!) https://c-ares.org/ I have created a PoC patch. https://github.com/mame/ruby/commit/547806146993bbc25984011d423dcc0f913b211c By applying this patch, we can interrupt `Addrinfo.getaddrinfo` by Ctrl+C. ``` # cat /etc/resolv.conf nameserver 72.66.115.13 # ./local/bin/ruby -rsocket -e 'Addrinfo.getaddrinfo("www.ruby-lang.org", 80)' ^C-e:1:in `getaddrinfo': Interrupt from -e:1:in `<main>' ``` ## Discussion ### About c-ares According to the site of c-ares, some major tools including libcurl, Wireshark, and Apache Arrow are already using c-ares. In the language interpreter, node.js seems to be using c-ares. I am honestly not sure about the compatibility of c-ares with `getaddrinfo(3)`. I guess there is no major incompatibility because I have not experienced any name resolution problem of curl. @akr (who is the author and maintainer of Ruby's socket library) suggested to check if OS-specific name resolution, e.g., WINS on Windows, NIS on Solaris, etc., is supported. He also said that it may be acceptable even if they are not supported. Whether to bundle c-ares source code with ruby would require further discussion. If this proposal is accepted, then c-ares will become a de facto essential dependency for practical use, like gmp, in my opinion. Incidentally, node.js bundles c-ares: https://github.com/nodejs/node/tree/main/deps/cares ### Alternative approaches Recent glibc provides `getaddrinfo_a(3)` which performs asynchronous name resolution. However, this function has a fatal problem of being incompatible with `fork(2)`, which is heavily used in the Ruby ecosystem. In fact, the attempt to use `getaddrinfo_a(3)` (#17134) has been revert because it fails rails tests. (#17220) Another alternative is to have a worker pthread inside Ruby that calls getaddrinfo(3). Instead of calling getaddrinfo(3) directly, `Addrinfo.getaddrinfo` would ask the worker to resolve a name and wait for a response. This method should be able to implement cancellation. (Simply put, this means reimplementation of getaddrinfo_a(3) on our own, taking into account of `fork(2).) This has the advantages: not adding dependencies on external libraries and not having compatibility issues with `getaddrinfo(3)`. However, it is considerably more difficult to implement and maintain. An internal pthread may have a non-trivial impact on the execution efficiency and memory usage. Also, we may need to implement a mechanism to dynamically change the number of workers depending on the load. It would be ideal if we could try and evaluate both approaches. But my current impression is that using c-ares is the quickest and best compromise. ## Contribution wanted I have made it up to the PoC, but don't have much time to complete this. @naruse suggested me to create a ticket asking for contributions. Is anyone interested in this? * This patch changes `rsock_getaddrinfo` to accept a timeout argument. There are several places where Qnil is passed as a timeout (where I add `// TODO` in the PoC). We need to consider what timeout we should pass. * This cares only `getaddrinfo`, but we also need to care `getnameinfo` (and something else if any). There may be some issues I'm not aware of. * I have not yet tested this PoC seriously. It would be great if we could evaluate it with some real apps. Also, it would be great to hear from someone who knows more about c-ares. -- https://bugs.ruby-lang.org/

6 15

[ruby-core:115082] [Ruby master Bug#19932] Regexp o modifier freeze interpolation between method calls
by noraj-acceis (Alexandre ZANNI) 18 Oct '23

18 Oct '23

Issue #19932 has been reported by noraj-acceis (Alexandre ZANNI). ---------------------------------------- Bug #19932: Regexp o modifier freeze interpolation between method calls https://bugs.ruby-lang.org/issues/19932 * Author: noraj-acceis (Alexandre ZANNI) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Taken the following PoC: ```ruby def poc(regexp) hs = [ 'azerty', 'azertyui', 'azertyuiop' ] out = [] hs.each do |h| out << h if /#{regexp}/.match?('azerty') end out end p poc('az') p poc('wxc') ``` I have the following output: ```ruby ["azerty", "azertyui", "azertyuiop"] [] ``` Now because the regexp never change inside the method once set I could add the `o` modifier so that `#{}` interpolation is computed only once. So the PoC becomes: ```ruby def poc(regexp) hs = [ 'azerty', 'azertyui', 'azertyuiop' ] out = [] hs.each do |h| out << h if /#{regexp}/o.match?('azerty') end out end p poc('az') p poc('wxc') ``` Output: ``` ["azerty", "azertyui", "azertyuiop"] ["azerty", "azertyui", "azertyuiop"] ``` So each future call to the method will be equals to the result of the first call because the regexp is "frozen". I was expecting the regexp to be "frozen" only for the current call in an effort of performance but instead it's "frozen" for future calls too. Another example of usage of `o` modifier here: https://learnbyexample.github.io/Ruby_Regexp/modifiers.html#o-modifier So is that a side effect or is that the expected behavior? -- https://bugs.ruby-lang.org/

2 4

[ruby-core:115070] [Ruby master Feature#19930] [Documentation] class Regexp: Character Classes ranges
by noraj-acceis (Alexandre ZANNI) 17 Oct '23

17 Oct '23

Issue #19930 has been reported by noraj-acceis (Alexandre ZANNI). ---------------------------------------- Feature #19930: [Documentation] class Regexp: Character Classes ranges https://bugs.ruby-lang.org/issues/19930 * Author: noraj-acceis (Alexandre ZANNI) * Status: Open * Priority: Normal ---------------------------------------- cf. https://ruby-doc.org/3.2.2/Regexp.html#class-Regexp-label-Character+Classes > POSIX bracket expressions are also similar to character classes. They provide a portable alternative to the above, with the added benefit that they encompass non-ASCII characters. For instance, /\d/ matches only the ASCII decimal digits (0-9); whereas /[[:digit:]]/ matches any character in the Unicode Nd category. Reading this description, we globally expect that metacharacters are ASCII only and that POSIX _bracket expressions_ are Unicode aware. But as _bracket expressions_ are POSIX compliant, for example `[:xdigit:]` use only ASCII range `[A-Fa-f0-9]` and not the `Hex_Digit` Unicode property that is also including the Halfwidth and Fullwidth Forms Number Decimal like `０` (U+FF10, FULLWIDTH DIGIT ZERO). So the above description is confusing as we would expect [[:xdigit:]]` to _encompass non-ASCII characters_ too. On the contrary `[:space:]` will look for `[\p{Z}\t\r\n\v\f]` (`\s` plus `\p{Z}` (Separator)) while the description is talking only about `[:blank:], newline, carriage return`. My point is, in the end, that it's hard to determine what to expect as ranges for character classes while reading the Ruby Regexp documentation alone. To know what is the exact behavior I'll have to read the source code or at least reading the POSIX spec. My feature request is about adding a comparison table like the one on https://www.regular-expressions.info/posixbrackets.html (for Java) with: the POSIX bracket expression, the description, the ASCII exact range, the Unicode exact range, the shorthand metacharacter (ASCII), the long escape sequence (Unicode). So we could know precisely what to expect by reading the doc. ---Files-------------------------------- Screenshot_20231017_154208.png (145 KB) -- https://bugs.ruby-lang.org/

1 0

[ruby-core:114250] [Ruby master Bug#19778] mkmf.rb pkg_config() interaction with RbConfig::CONFIG["cflags"]
by rhenium (Kazuki Yamaguchi) 17 Oct '23

17 Oct '23

Issue #19778 has been reported by rhenium (Kazuki Yamaguchi). ---------------------------------------- Bug #19778: mkmf.rb pkg_config() interaction with RbConfig::CONFIG["cflags"] https://bugs.ruby-lang.org/issues/19778 * Author: rhenium (Kazuki Yamaguchi) * Status: Open * Priority: Normal * ruby -v: ruby 3.3.0dev (2023-07-21T09:38:29Z master 22f9735587) [x86_64-linux] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- (This was first reported at https://github.com/ruby/openssl/issues/650: The extension's build breaks in a strange way if `RbConfig::CONFIG["*flags"]` contains the path of an OpenSSL installation but `pkg-config` returns the path of a different OpenSSL installation) Commit commit:097c3e9cbbf23718371f08c24b2d2297b039f63f ("mkmf.rb: -I flags to $INCFLAGS", Ruby 2.2) changed how mkmf's `pkg_config()` handles the result of the `pkg-config` command. It now stores the `-I` flags in $INCFLAGS and others in $CFLAGS. mkmf generates a Makefile that compiles source files with: $(CC) $(INCFLAGS) $(CPPFLAGS) $(CFLAGS) and link the final library with: $(LDSHARED) -o $@ $(OBJS) $(LIBPATH) $(DLDFLAGS) $(LOCAL_LIBS) $(LIBS) This "new" behavior of `pkg_config()` is problematic when `RbConfig::CONFIG["{C,CPP}FLAGS"]` also provide `-I` flags and `RbConfig::CONFIG["LDFLAGS"]` provides the matching `-L` flags -- for example, if Ruby is compiled with `./configure --with-opt-dir=<dir>`. This would end up with compiling sources with [...] -I<from pkg-config> -I<from RbConfig> [...] and then linking with [...] -L<from RbConfig> -L<from pkg-config> [...] This doesn't seem right. I don't know which should come earlier, but the order should be consistent. The commit in question clearly describes the change in the commit message, but it doesn't have a linked issue. What is it intended for? Also, what is $INCFLAGS? On the other hand, `dir_config()` would prepend `-I` flags to $CPPFLAGS and `-L` flags to $LIBPATH, so it doesn't have issues with flags from `RbConfig`, albeit in a differently way. -- https://bugs.ruby-lang.org/

5 4

[ruby-core:114826] [Ruby master Bug#19894] Memory leak in complemented method entries
by peterzhu2118 (Peter Zhu) 17 Oct '23

17 Oct '23

Issue #19894 has been reported by peterzhu2118 (Peter Zhu). ---------------------------------------- Bug #19894: Memory leak in complemented method entries https://bugs.ruby-lang.org/issues/19894 * Author: peterzhu2118 (Peter Zhu) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: REQUIRED ---------------------------------------- GitHub pull request: https://github.com/ruby/ruby/pull/8481 When a copy of a complemented method entry is created, there are two issues: 1. IMEMO_FL_USER3 is not copied, so the complemented status is not copied over. 2. In rb_method_entry_clone we increment both alias_count and complemented_count. However, when we free the method entry in rb_method_definition_release, we only decrement one of the two counters, resulting in the rb_method_definition_t being leaked. The following script reproduces this issue: ```ruby 10.times do 20_000.times do $c = Class.new do def foo; end end $m = Module.new do refine $c do def foo; end end end Class.new do using $m def initialize o = $c.new o.method(:foo).unbind end end.new end puts `ps -o rss= -p #{$$}` end ``` Before this fix: ``` 17328 19248 21408 23296 25600 27408 29424 31520 33728 35664 ``` After this fix: ``` 16240 17680 18800 19744 20640 21840 22896 24336 25280 26096 ``` -- https://bugs.ruby-lang.org/

3 2

[ruby-core:114992] [Ruby master Bug#19917] Segmentation fault or lost objects when using Ractor.select with moved exceptions
by toy (Ivan Kuchin) 15 Oct '23

15 Oct '23

Issue #19917 has been reported by toy (Ivan Kuchin). ---------------------------------------- Bug #19917: Segmentation fault or lost objects when using Ractor.select with moved exceptions https://bugs.ruby-lang.org/issues/19917 * Author: toy (Ivan Kuchin) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- I stumbled upon loss of messages with exceptions or even Segmentation fault during transfer of exception from Ractor when moving the exception. In versions 3.0 and 3.1 I saw only loss of messages, in 3.2 also Segmentation fault. Loss of messages happens only when using `Ractor.select` (not `Ractor#take`), but segmentation fault happens also using `Ractor#take`. ```ruby 100.times do ractor_count = 10 ractors = Array.new(ractor_count) do |i| Ractor.new do begin raise 'foo' rescue => e # It is not possible to move exception without duplicating, but also # without it showing bactrace errors in 3.0 and is empty in 3.1+ e = Marshal.load(Marshal.dump(e)) Ractor.yield e, move: true end 'message got lost' end end ractor_count.times do ractor, result = Ractor.select(*ractors) p result ractors.delete(ractor) end end ``` In 3.0 and 3.1 will contain mix of `#<RuntimeError: foo>` and `"message got lost"`, in 3.2 may contain both and cause Segmentation fault. Excerpt of output (full output attached) from container running ruby:3.2 (`ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux]`): ``` … #<RuntimeError: RuntimeError> "message got lost" #<RuntimeError: RuntimeError> #<RuntimeError: RuntimeError> #<RuntimeError: RuntimeError> #<RuntimeError: RuntimeError> #<RuntimeError: RuntimeError> #<RuntimeError: RuntimeError> #<RuntimeError: RuntimeError> #<RuntimeError: RuntimeError> #<RuntimeError: RuntimeError> #<RuntimeError: RuntimeError> #<RuntimeError: RuntimeError> #<RuntimeError: RuntimeError> #<RuntimeError: RuntimeError> #<RuntimeError: RuntimeError> #<RuntimeError: RuntimeError> #<RuntimeError: RuntimeError> #<RuntimeError: RuntimeError> #<RuntimeError: RuntimeError> <internal:marshal>:34: [BUG] Segmentation fault at 0x0000000000000000 ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux] -- Control frame information ----------------------------------------------- c:0004 p:0006 s:0019 e:000018 METHOD <internal:marshal>:34 c:0003 p:0022 s:0011 e:000010 RESCUE (irb):11 c:0002 p:0007 s:0007 e:000006 BLOCK (irb):6 [FINISH] c:0001 p:---- s:0003 e:000002 DUMMY [FINISH] -- Ruby level backtrace information ---------------------------------------- (irb):6:in `block (3 levels) in <top (required)>' (irb):11:in `rescue in block (3 levels) in <top (required)>' <internal:marshal>:34:in `load' -- Machine register context ------------------------------------------------ RIP: 0x00007fabc1d64490 RBP: 0x0000000000000000 RSP: 0x00007fabbc71b0c0 RAX: 0x0000000000000003 RBX: 0x00007fabbcebee88 RCX: 0x000000000000015f RDX: 0x0000556127db90f0 RDI: 0x000000000000015f RSI: 0x00007fabc2068628 R8: 0x0000000000000001 R9: 0x0000000000000020 R10: 0x00005561285633d0 R11: 0x0000000000000001 R12: 0x0000556127dbb710 R13: 0x00007fabc0601be0 R14: 0x0000000000000018 R15: 0x0000000000000000 EFL: 0x0000000000010206 … ``` ---Files-------------------------------- segmentation-fault.txt (27.8 KB) -- https://bugs.ruby-lang.org/

2 2

[ruby-core:115054] [Ruby master Feature#18822] Ruby lack a proper method to percent-encode strings for URIs (RFC 3986)
by noraj (Alexandre ZANNI) 14 Oct '23

14 Oct '23

Issue #18822 has been updated by noraj (Alexandre ZANNI). I just want to complete what was said before. URI.escape and URI.unescape were deprecated but they were replaced by URI::Parser.new.escape and URI::Parser.new.unescape that implements RFC 2396. In fact this is calling URI::RFC2396_Parser.escape and URI::RFC2396_Parser.unescape. But it's not useless since RFC 2396 was a Draft Standard and was obsoleted and updated by RFC 3986 which is an Internet Standard as CGI.escapeURIComponent and CGI.unescapeURIComponent implements RFC 3986. ---------------------------------------- Feature #18822: Ruby lack a proper method to percent-encode strings for URIs (RFC 3986) https://bugs.ruby-lang.org/issues/18822#change-104929 * Author: byroot (Jean Boussier) * Status: Closed * Priority: Normal ---------------------------------------- ### Context There are two fairly similar encoding methods that are often confused. `application/x-www-form-urlencoded` which is how form data is encoded, and "percent-encoding" as defined by [RFC 3986](https://www.rfc-editor.org/rfc/rfc3986). AFAIK, the only way they differ is that "form encoding" escape space characters as `+`, and RFC 3986 escape them as `%20`. Most of the time it doesn't matter, but sometimes it does. ### Ruby form and URL escape methods - `URI.escape(" ") # => "%20"` but it was deprecated and removed (in 3.0 ?). - `ERB::Util.url_encode(" ") # => "%20"` but it's implemented with a `gsub` and isn't very performant. It's also awkward to have to reach for `ERB` - `CGI.escape(" ") # => "+"` - `URI.encode_www_form_component(" ") # => "+"` ### Unescape methods For unescaping, it's even more of a clear cut since `URI.unescape` was removed. So there's no available method that won't treat an unescaped `+` as simply `+`. e.g. in Javascript: `decodeURIComponent("foo+bar") #=> "foo+bar"`. If one were to use `CGI.unescape`, the string might be improperly decoded: `GI.unescape("foo+bar") #=> "foo bar"`. ### Other languages - Javascript `encodeURI` and `encodeURIComponent` use `%20`. - PHP has `urlencode` using `+` and `rawurlencode` using `%20`. - Python has `urllib.parse.quote` using `%20` and `urllib.parse.quote_plus` using `+`. ### Proposal Since `CGI` already have a very performant encoder for `application/x-www-form-urlencoded`, I think it would make sense that it would provide another method for RFC3986. I propose: - `CGI.url_encode(" ") # => "%20"` - Or `CGI.encode_url`. - Alias `CGI.escape` as `GCI.encode_www_form_component` - Clarify the documentation of `CGI.escape`. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:115053] [Ruby master Feature#16294] Make MatchData frozen and forbid MatchData.allocate
by Eregon (Benoit Daloze) 14 Oct '23

14 Oct '23

Issue #16294 has been updated by Eregon (Benoit Daloze). nobu (Nobuyoshi Nakada) wrote in #note-8: > At the latest dev-meeting, there was a suggestion that whereas `MatchData` cannot be dumped, `Regexp.allocate` might be used by serializer libraries. That does not seem used in practice. TruffleRuby already does not define `Regexp.allocate`, and there seems to be no compatibility issue with that except some specs. `Regexp.new` (or `eval`) is good enough for deserializing regexps. ---------------------------------------- Feature #16294: Make MatchData frozen and forbid MatchData.allocate https://bugs.ruby-lang.org/issues/16294#change-104928 * Author: Eregon (Benoit Daloze) * Status: Closed * Priority: Normal ---------------------------------------- Currently, `MatchData.allocate` is allowed, but almost every MatchData method called on it `raise TypeError, 'uninitialized Match'`. I think MatchData should be frozen, none of its internal fields are mutable and I don't see any use case for storing instance variables on it. Once frozen, we can implement MatchData#dup and #clone as just `return self`, and we don't need to check for the uninitialized case. And Marshal can have special treatment to create an initialized MatchData directly. My main motivation is looking at the code in TruffleRuby required to implement `MatchData.allocate` and check if it's initialized in so many places: https://github.com/oracle/truffleruby/pull/1792/files Thoughts? Anyone against? cc @alanwu -- https://bugs.ruby-lang.org/

1 0

[ruby-core:115052] [Ruby master Misc#18984] Doc for Range#size for Float/Rational does not make sense
by kyanagi (Kouhei Yanagita) 14 Oct '23

14 Oct '23

Issue #18984 has been updated by kyanagi (Kouhei Yanagita). I created a PR: https://github.com/ruby/ruby/pull/8663 What should we do about `Range#count`? ``` (..0).count # => Infinity (nil..nil).count # => Infinity ``` ---------------------------------------- Misc #18984: Doc for Range#size for Float/Rational does not make sense https://bugs.ruby-lang.org/issues/18984#change-104927 * Author: masasakano (Masa Sakano) * Status: Open * Priority: Normal ---------------------------------------- When `Range` consists of any Numeric, according to [Official docs for Ruby-3.1.2](https://ruby-doc.org/core-3.1.2/Range.html#method-i-size), `Range#size` should, > Returns the count of elements in self if both begin and end values are numeric; Indeed, when `Range` consists of only `Integer`, this makes sense. However, when the begin value of `Range` is a `Float` or `Rational`, "*the count of elements*" does not make sense. Such Ranges are not iteratable, suggesting there are no such things as "elements": ```ruby (0.51..5.quo(2)).each{} # => TypeError ``` Yet, `Range#size` of such Ranges returns an Integer ```ruby (0.51..5.quo(2)).size # => 2 ``` It seems both begin and end values of a Range are rounded (`Numeric#round`) to the nearest Integer before `Range#size` is calculated. If this is the specification, I suggest it should be clearly stated in the [Official docs](https://ruby-doc.org/core-3.1.2/Range.html#method-i-size), avoiding the confusing expression "the count of elements", because "elements" are not unambiguously defined for Range of Float/Rational. -- https://bugs.ruby-lang.org/

1 0