May 2024 - ruby-core - ml.ruby-lang.org

[ruby-core:117619] [Ruby master Bug#20438] String format "%\n" and "%\0" does not raise format error
by tompng (tomoya ishida) 28 May '24

28 May '24

Issue #20438 has been reported by tompng (tomoya ishida). ---------------------------------------- Bug #20438: String format "%\n" and "%\0" does not raise format error https://bugs.ruby-lang.org/issues/20438 * Author: tompng (tomoya ishida) * Status: Open * ruby -v: ruby 3.3.0 (2023-12-25 revision 5124f9ac75) +YJIT +MN [arm64-darwin22] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- `"%" % 1` raises `incomplete format specifier; use %% (double %) instead` `"%=" % 1` raises `malformed format string - %=`. But `"%\n" % 1` `"%\0" % 1` does not raise error. In `sprintf.c`, `\n` and `\0` are explicitly accepted. Is this expected? Some other language are: Perl: Warns `Invalid conversion in printf`. Just prints. `"%d% " → "1% "` Python: Error `ValueError: unsupported format character '?' (0xa)` with `print("%\n" % 123)` PHP: Error `Unknown format specifier` with `sprintf("%\n", 3)` C, C++: Warns `incomplete format specifier`. `"%\n" → "\n"`, `"%" → ""`, `"% " → ""` `sprintf("%f%\n",x)` and `"%f%\n" % x` is used in some codes https://github.com/search?q=language%3ARuby+%22f%25%5Cn%22&type=code -- https://bugs.ruby-lang.org/

2 2

[ruby-core:118037] [Ruby master Bug#20512] Order of magnitude performance differenfce in single character slicing UTF-8 strings before and after length method is executed
by giner (Stanislav German-Evtushenko) 28 May '24

28 May '24

Issue #20512 has been reported by giner (Stanislav German-Evtushenko). ---------------------------------------- Bug #20512: Order of magnitude performance differenfce in single character slicing UTF-8 strings before and after length method is executed https://bugs.ruby-lang.org/issues/20512 * Author: giner (Stanislav German-Evtushenko) * Status: Open * ruby -v: ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- Slicing of a single character of UTF-8 string becomes ~15 times faster after method "length" is executed on the string. ```ruby # Single byte symbols letters = ("a".."z").to_a length = 100000 str = length.times.map{letters[rand(26)]}.join # Slow start = Time.now length.times{|i| str[i]} puts Time.now - start # 0.169156201 str.length # performance hack # Fast start = Time.now length.times{|i| str[i]} puts Time.now - start # 0.009883919 # UTF-8 Symbols letters = ("а".."я").to_a length = 10000 str = length.times.map{letters[rand(26)]}.join # Slow start = Time.now length.times{|i| str[i]} puts Time.now - start # 0.326204007 str.length # performance hack # Fast start = Time.now length.times{|i| str[i]} puts Time.now - start # 0.016943093 ``` -- https://bugs.ruby-lang.org/

2 1

[ruby-core:117469] [Ruby master Feature#20415] Precompute literal String hash code during compilation
by byroot (Jean Boussier) 28 May '24

28 May '24

Issue #20415 has been reported by byroot (Jean Boussier). ---------------------------------------- Feature #20415: Precompute literal String hash code during compilation https://bugs.ruby-lang.org/issues/20415 * Author: byroot (Jean Boussier) * Status: Open ---------------------------------------- I worked on a proof of concept with @etienne which I think has some potential, but I'm looking for feedback on what would be the best implementation. The proof of concept is here: https://github.com/Shopify/ruby/pull/553 ### Idea Most string literals are relatively short, hence embedded, and have some wasted bytes at the end of their slot. We could use that wasted space to store the string hash. The goal being to make **looking up a literal String key in a hash, as fast as a Symbol key**. The goal isn't to memoize the hash code of all strings, but to **only selectively precompute the hash code of literal strings in the compiler**. The compiler could even selectively do this when we literal string is used to lookup a hash (`opt_aref`). Here's the benchmark we used: ```ruby hash = 10.times.to_h do |i| [i, i] end dyn_sym = "dynamic_symbol".to_sym hash[:some_symbol] = 1 hash[dyn_sym] = 1 hash["small"] = 2 hash["frozen_string_literal"] = 2 Benchmark.ips do |x| x.report("symbol") { hash[:some_symbol] } x.report("dyn_symbol") { hash[:some_symbol] } x.report("small_lit") { hash["small"] } x.report("frozen_lit") { hash["frozen_string_literal"] } x.compare!(order: :baseline) end ``` On Ruby 3.3.0, looking up a String key is a bit slower based on the key size: ``` Calculating ------------------------------------- symbol 24.175M (± 1.7%) i/s - 122.002M in 5.048306s dyn_symbol 24.345M (± 1.6%) i/s - 122.019M in 5.013400s small_lit 21.252M (± 2.1%) i/s - 107.744M in 5.072042s frozen_lit 20.095M (± 1.3%) i/s - 100.489M in 5.001681s Comparison: symbol: 24174848.1 i/s dyn_symbol: 24345476.9 i/s - same-ish: difference falls within error small_lit: 21252403.2 i/s - 1.14x slower frozen_lit: 20094766.0 i/s - 1.20x slower ``` With the proof of concept performance is pretty much identical: ``` Calculating ------------------------------------- symbol 23.528M (± 6.9%) i/s - 117.584M in 5.033231s dyn_symbol 23.777M (± 4.7%) i/s - 120.231M in 5.071734s small_lit 23.066M (± 2.9%) i/s - 115.376M in 5.006947s frozen_lit 22.729M (± 1.1%) i/s - 115.693M in 5.090700s Comparison: symbol: 23527823.6 i/s dyn_symbol: 23776757.8 i/s - same-ish: difference falls within error small_lit: 23065535.3 i/s - same-ish: difference falls within error frozen_lit: 22729351.6 i/s - same-ish: difference falls within error ``` ### Possible implementation The reason I'm opening this issue early is to get feedback on which would be the best implementation. #### Store hashcode after the string terminator Right now the proof of concept simply stores the `st_index_t` after the string null terminator, and only when the string is embedded and as enough left over space. Strings with a precomputed hash are marked with an user flag. Pros: - Very simple implementation, no need to change a lot of code, and very easy to strip out if we want to. - Doesn't use any extra memory. If the string doesn't have enough left over bytes, the optimization simply isn't applied. - The worst case overhead is a single `FL_TEST_RAW` in `rb_str_hash`. Cons: - The optimization won't apply to certain string sizes. e.g. strings between `17` and `23` bytes won't have a precomputed hash code. - Extracting the hash code requires some not so nice pointer arithmetic. #### Create another RString union Another possibility would be to add another entry in the `RString` struct union, such as we'd have: ```c struct RString { struct RBasic basic; long len; union { // ... existing members struct { st_index_t hash; char ary[1]; } embded_literal; } as; }; ``` Pros: - The optimization can now be applied to all string sizes. - The hashcode is always at the same offset and properly aligned. Cons: - Some strings would be bumped by one slot size, so would use marginally more memory. - Complexify the code base more, need to modify a lot more string related code (e.g. `RSTRING_PTR` and many others) - When compiling such string, if an equal string already exists in the `fstring` table, we'd need to replace it, we can't just mutate it in place to add the hashcode. ### Prior art [Feature #15331] is somewhat similar in its idea, but it does it lazily for all strings. Here it's much simpler because limited to string literals, which are the ones likely to be used as Hash keys, and the overhead is on compilation, not runtime (aside from a single flag check). So I think most of the caveats of that original implementation don't apply here. -- https://bugs.ruby-lang.org/

4 9

[ruby-core:117928] [Ruby master Feature#20497] Tempfile.create_io
by akr (Akira Tanaka) 28 May '24

28 May '24

Issue #20497 has been reported by akr (Akira Tanaka). ---------------------------------------- Feature #20497: Tempfile.create_io https://bugs.ruby-lang.org/issues/20497 * Author: akr (Akira Tanaka) * Status: Open ---------------------------------------- I propose Tempfile.create_io. It is similar to Tempfile.create but the actual file is unlinked before returning the method. https://github.com/ruby/ruby/pull/10803 Purpose: Sometimes, applications need a temporary file but it is not required to access via a pathname. In this case, the created file can be unlinked just after temporary file creation. This removes the obligation of removing the file from applications. So, Tempfile.create_io is easier to use than Tempfile.create. Example: ``` tmpio = Tempfile.create_io # => #<IO:fd 5> tmpio.class # => IO tmpio.path # => nil tmpio.stat.mode.to_s(8) # => "100600" tmpio.puts "foo" tmpio.rewind tmpio.read # => "foo\n" tmpio.close ``` Portability: This feature (unlink just after file creation) is supported on Unix for a long time. Linux 3.11 has O_TMPFILE to create an unnamed file. The current implementation uses it. I heard that it is possible on Windows. (O_SHARE_DELETE?) I'm not sure how to use it. -- https://bugs.ruby-lang.org/

5 7

[ruby-core:113410] [Ruby master Bug#19631] module_eval does not propulate absolute_path for Kernel.caller_locations
by daveola (David Stellar) 27 May '24

27 May '24

Issue #19631 has been reported by daveola (David Stellar). ---------------------------------------- Bug #19631: module_eval does not propulate absolute_path for Kernel.caller_locations https://bugs.ruby-lang.org/issues/19631 * Author: daveola (David Stellar) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- I am using module_eval and noticing that since ruby 3.2 the Kernel locations do not have absolute_path for any of the eval code, though the path is available. This is a regression since at least ruby 3.0 which still works. I am on 3.2.2, and here is some sample code: ---- class Script < Module script = %q{ def self.locations Kernel.caller_locations.each { |loc| puts "LOCATION: #{loc}" puts "ABSPATH: #{loc.absolute_path}" puts "PATH: #{loc.path}" } end self.locations } module_eval(script, "/this/is/my/path", 0) end ------ The output for the code at the top of the caller locations (inside the module_eval) is: LOCATION: /this/is/my/path:9:in `<class:Script>' ABSPATH: PATH: /this/is/my/path But the absolute_path should have the correct path data as well -- https://bugs.ruby-lang.org/

2 1

[ruby-core:118029] Provide utf-8 sorting for ruby next
by d_rems＠yahoo.com 27 May '24

27 May '24

Since it has been declared that frozen_string_literal will print warnings in Ruby 3.4. and will be mandatory in the future I must warn, that ruby still doesn't have built in method for sorting by UTF-8 characters. Most of string methods work OK with UTF-8 strings, but not sort. I've been using sort_alphabetical gem [ https://github.com/grosser/sort_alphabetical ] for a long time and it works good for me. Problem is that it depends on unicode_utils gem [ https://github.com/lang/unicode_utils ] which looks like a dead repository, since repository owner has not been active for a decade. And unicode_utils use a lot of code which is not complied with frozen_string_literal. I am proposing to include UTF-8 sort (ex. sort_by_utf) method into next ruby version or at least include (and correct) those two gems in a future Ruby distribution. Thanks by TheR

2 1

[ruby-core:118007] [Ruby master Misc#20507] Allow C extensions to be compiled cleanly with the `-Wsign-conversion` warning option
by mdalessio (Mike Dalessio) 27 May '24

27 May '24

Issue #20507 has been reported by mdalessio (Mike Dalessio). ---------------------------------------- Misc #20507: Allow C extensions to be compiled cleanly with the `-Wsign-conversion` warning option https://bugs.ruby-lang.org/issues/20507 * Author: mdalessio (Mike Dalessio) * Status: Open ---------------------------------------- As a maintainer of several C extensions, I like to compile my code with some of the common [Warning Options](https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html) to help maintain code quality and safety. One of the options that is currently very difficult to use is `-Wsign-conversion`, because some of the Ruby public header files contain code that will generate warnings. So many warnings are printed when compiling against Ruby's header files that it's hard to see any warnings from my code. As an example, compiling just one file in Nokogiri with this flag enabled will print: ``` .../include/ruby-3.4.0+0/ruby/internal/special_consts.h: In function ‘RB_TEST’: .../include/ruby-3.4.0+0/ruby/internal/special_consts.h:159:16: warning: unsigned conversion from ‘int’ to ‘VALUE’ {aka ‘long unsigned int’} changes value from ‘-5’ to ‘18446744073709551611’ [-Wsign-conversion] 159 | return obj & ~RUBY_Qnil; | ^ .../include/ruby-3.4.0+0/ruby/internal/special_consts.h: In function ‘RB_NIL_OR_UNDEF_P’: .../include/ruby-3.4.0+0/ruby/internal/special_consts.h:229:24: warning: unsigned conversion from ‘int’ to ‘VALUE’ {aka ‘const long unsigned int’} changes value from ‘-33’ to ‘18446744073709551583’ [-Wsign-conversion] 229 | const VALUE mask = ~(RUBY_Qundef ^ RUBY_Qnil); | ^ .../include/ruby-3.4.0+0/ruby/internal/arithmetic/long.h: In function ‘RB_INT2FIX’: .../include/ruby-3.4.0+0/ruby/internal/arithmetic/long.h:117:29: warning: conversion to ‘long unsigned int’ from ‘long int’ may change the sign of the result [-Wsign-conversion] 117 | const unsigned long j = i; | ^ .../include/ruby-3.4.0+0/ruby/internal/arithmetic/long.h:119:29: warning: conversion to ‘long int’ from ‘long unsigned int’ may change the sign of the result [-Wsign-conversion] 119 | const long l = k; | ^ .../include/ruby-3.4.0+0/ruby/internal/arithmetic/long.h:121:29: warning: conversion to ‘VALUE’ {aka ‘const long unsigned int’} from ‘long int’ may change the sign of the result [-Wsign-conversion] 121 | const VALUE n = m; | ^ .../include/ruby-3.4.0+0/ruby/internal/arithmetic/long.h: In function ‘rbimpl_fix2long_by_idiv’: .../include/ruby-3.4.0+0/ruby/internal/arithmetic/long.h:169:28: warning: conversion to ‘long int’ from ‘VALUE’ {aka ‘long unsigned int’} may change the sign of the result [-Wsign-conversion] 169 | const SIGNED_VALUE y = x - RUBY_FIXNUM_FLAG; | ^ .../include/ruby-3.4.0+0/ruby/internal/arithmetic/long.h: In function ‘rbimpl_fix2long_by_shift’: .../include/ruby-3.4.0+0/ruby/internal/arithmetic/long.h:196:28: warning: conversion to ‘long int’ from ‘VALUE’ {aka ‘long unsigned int’} may change the sign of the result [-Wsign-conversion] 196 | const SIGNED_VALUE y = x; | ^ .../include/ruby-3.4.0+0/ruby/internal/arithmetic/long.h: In function ‘rb_fix2ulong’: .../include/ruby-3.4.0+0/ruby/internal/arithmetic/long.h:255:12: warning: conversion to ‘long unsigned int’ from ‘long int’ may change the sign of the result [-Wsign-conversion] 255 | return rb_fix2long(x); | ^~~~~~~~~~~~~~ .../include/ruby-3.4.0+0/ruby/internal/arithmetic/long.h: In function ‘rb_ulong2num_inline’: .../include/ruby-3.4.0+0/ruby/internal/arithmetic/long.h:326:28: warning: conversion to ‘long int’ from ‘long unsigned int’ may change the sign of the result [-Wsign-conversion] 326 | return RB_LONG2FIX(v); | ^ .../include/ruby-3.4.0+0/ruby/internal/arithmetic/long_long.h: In function ‘rb_num2ull_inline’: .../include/ruby-3.4.0+0/ruby/internal/arithmetic/long.h:53:22: warning: conversion to ‘long long unsigned int’ from ‘long int’ may change the sign of the result [-Wsign-conversion] 53 | #define RB_FIX2LONG rb_fix2long /**< @alias{rb_fix2long} */ .../include/ruby-3.4.0+0/ruby/internal/arithmetic/long_long.h:130:16: note: in expansion of macro ‘RB_FIX2LONG’ 130 | return RB_FIX2LONG(x); | ^~~~~~~~~~~ In file included from .../include/ruby-3.4.0+0/ruby/internal/arithmetic.h:37: .../include/ruby-3.4.0+0/ruby/internal/arithmetic/st_data_t.h: In function ‘RB_ST2FIX’: .../include/ruby-3.4.0+0/ruby/internal/arithmetic/st_data_t.h:61:22: warning: conversion to ‘long int’ from ‘st_data_t’ {aka ‘long unsigned int’} may change the sign of the result [-Wsign-conversion] 61 | SIGNED_VALUE x = i; | ^ .../include/ruby-3.4.0+0/ruby/internal/arithmetic/st_data_t.h:72:24: warning: conversion to ‘long int’ from ‘long unsigned int’ may change the sign of the result [-Wsign-conversion] 72 | return RB_LONG2FIX(y); | ^ In file included from .../include/ruby-3.4.0+0/ruby/ruby.h:42: .../include/ruby-3.4.0+0/ruby/internal/memory.h: In function ‘rb_alloc_tmp_buffer2’: .../include/ruby-3.4.0+0/ruby/internal/memory.h:646:56: warning: conversion to ‘size_t’ {aka ‘long unsigned int’} from ‘long int’ may change the sign of the result [-Wsign-conversion] 646 | const size_t total_size = rbimpl_size_mul_or_raise(count, elsize); | ^~~~~ In file included from .../include/ruby-3.4.0+0/ruby/internal/encoding/ctype.h:27, from .../include/ruby-3.4.0+0/ruby/encoding.h:22, from ../../../../ext/nokogiri/nokogiri.h:79: .../include/ruby-3.4.0+0/ruby/internal/encoding/encoding.h: In function ‘RB_ENCODING_SET_INLINED’: .../include/ruby-3.4.0+0/ruby/internal/encoding/encoding.h:83:28: warning: conversion to ‘VALUE’ {aka ‘long unsigned int’} from ‘int’ may change the sign of the result [-Wsign-conversion] 83 | VALUE f = /* upcast */ encindex; | ^~~~~~~~ ``` Compiling all 35 files in Nokogiri's C extension means these warnings are repeated another 34 times in a full build. This makes it difficult to see any conversion errors in my own code. I'm about to open a pull request which adds casts where there are intentional, known-safe integer conversions occurring that are triggering these warnings. If merged, then C extension owners can add this to their `extconf.rb`: ``` append_cflags("-Wconversion") ``` and then any warnings generated are from the C extension code, and not from Ruby's header files. -- https://bugs.ruby-lang.org/

3 4

[ruby-core:117921] [Ruby master Bug#20495] Running "make clean" deletes critical "coroutine/amd64/Context.S" file and causes "make" to fail
by fallwith (James Bunch) 27 May '24

27 May '24

Issue #20495 has been reported by fallwith (James Bunch). ---------------------------------------- Bug #20495: Running "make clean" deletes critical "coroutine/amd64/Context.S" file and causes "make" to fail https://bugs.ruby-lang.org/issues/20495 * Author: fallwith (James Bunch) * Status: Open * ruby -v: 3.4.0-preview1 * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- When attempting to build v3.4.0-preview1 from source, we discovered that `make clean` will delete `coroutine/amd64/Context.S`, and then neither `./configure` nor `make` will replace that file. When that file is missing, `make` will fail. `coroutine/amd64/Context.S` is included in the .tar.gz file, and `make clean` deletes it without leaving a solution for recreating it. ``` $ tar xzf ruby-3.4.0-preview1.tar.gz $ cd ruby-3.4.0-preview1 $ ll coroutine/amd64 .rw-r--r--@ 2.2k 501 20 2024-05-15 21:54 Context.h .rw-r--r--@ 1.1k 501 20 2024-05-15 21:54 Context.S $ ./configure $ make clean $ ll coroutine/amd64 .rw-r--r--@ 2.2k 501 20 2024-05-15 21:54 Context.h $ make (lots of healthy output here) compiling yjit.c building Rust YJIT (release mode) touch yjit/target/release/libyjit.a partial linking yjit/target/release/libyjit.a into yjit/target/release/libyjit.o make: *** No rule to make target `coroutine/amd64/Context.S', needed by `coroutine/amd64/Context.o'. Stop. ``` -- https://bugs.ruby-lang.org/

7 6

[ruby-core:118015] [Ruby master Bug#20510] File.open mode argument inconsistency
by akr (Akira Tanaka) 25 May '24

25 May '24

Issue #20510 has been reported by akr (Akira Tanaka). ---------------------------------------- Bug #20510: File.open mode argument inconsistency https://bugs.ruby-lang.org/issues/20510 * Author: akr (Akira Tanaka) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- I found that `File.new(1, mode: File::WRONLY, path: "foo")` succeeds but `File.new(1, File::WRONLY, path: "foo")` fails. It is inconsistent. I think they should both succeed. ``` % ./ruby -e 'p File.new(1, mode: File::WRONLY, path: "foo")' #<File:foo> % ./ruby -e 'p File.new(1, File::WRONLY, path: "foo")' -e:1:in 'File#initialize': no implicit conversion of Integer into String (TypeError) p File.new(1, File::WRONLY, path: "foo") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ from -e:1:in 'IO.new' from -e:1:in '<main>' % ./ruby -v ruby 3.4.0dev (2024-05-25T02:25:56Z tempfile-create-io 99463a8400) [x86_64-linux] ``` -- https://bugs.ruby-lang.org/

2 1

[ruby-core:117368] [Ruby master Bug#20401] Duplicated when clause warning line number
by kddnewton (Kevin Newton) 24 May '24

24 May '24

Issue #20401 has been reported by kddnewton (Kevin Newton). ---------------------------------------- Bug #20401: Duplicated when clause warning line number https://bugs.ruby-lang.org/issues/20401 * Author: kddnewton (Kevin Newton) * Status: Open * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- When you have a duplicated when clause, you get a warning for it. For example: ```ruby case foo when :bar when :baz when :bar end ``` you get `warning: duplicated `when' clause with line 2 is ignored`. But the when clause that is ignored is the one on line 4, not line 2. It seems like it's warning for the wrong line. -- https://bugs.ruby-lang.org/

2 4