- ruby-core - ml.ruby-lang.org

[ruby-core:116413] [Ruby master Feature#16049] optimization for frozen dynamic string literals "#{exp}".dup and +"#{exp}"
by Dan0042 (Daniel DeLorme) 24 Jan '24

24 Jan '24

Issue #16049 has been updated by Dan0042 (Daniel DeLorme). Dynamic string literals are not longer frozen, this can be closed. (Would it be possible for the author to close their own ticket?) ---------------------------------------- Feature #16049: optimization for frozen dynamic string literals "#{exp}".dup and +"#{exp}" https://bugs.ruby-lang.org/issues/16049#change-106435 * Author: Dan0042 (Daniel DeLorme) * Status: Open * Priority: Normal ---------------------------------------- When the decision was made that `frozen_string_literal: true` should also apply to dynamic string literals, it was mitigated with the following explanation: > "#{exp}".dup can be optimized so it won’t allocate extra objects like "...".freeze https://docs.google.com/document/u/1/d/1D0Eo5N7NE_unIySOKG9lVj_eyXf66BQPM4P… However that does not appear to be the case currently. Using this script that generates 100k String objects: ```ruby # frozen_string_literal: true def allocated GC.stat[:total_allocated_objects] end GC.disable c = ARGV.shift.to_sym x_eq_i = ARGV.shift=="i" x = "x" before = allocated 100_000.times do |i| x = i.to_s if x_eq_i case c when :normal then v = "#{x}" when :freeze then v = "#{x}".freeze when :minus then v = -"#{x}" when :dup then v = "#{x}".dup when :plus then v = +"#{x}" else raise end end after = allocated printf "%d\n", after-before ``` I get the following number of objects allocated ``` x= frozen_string_literal normal freeze minus dup plus 'x' false 100001 100001 100001 200001 100001 'x' true 100001 100001 100001 200001 200001 i false 200001 200001 299999 300001 200001 i true 200001 200001 200001 300001 300001 ``` We can see that `"#{x}".dup` and `+"#{x}"` allocate an extra object per iteration I also tested with `x = i.to_s` to see if deduplication of 100k identical strings was different from 100k different strings. In addition to the expected extra strings created by `i.to_s`, there's an additional 100k extra strings created for `-"#{i}"` when frozen_string_literal is false??? There may also be a memory leak here because while the number of objects increases by x3, memory usage increases by x4. Summary: I expected `"#{v}".dup` and `+"#{v}"` to behave the same regardless of frozen_string_literal (and optimize down to just one allocation) I expected `"#{v}".freeze` and `-"#{v}"` to behave the same regardless of frozen_string_literal (and optimize down to just one allocation) but they do not. I think they should. It would be nice. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:116393] [Ruby master Feature#18576] Rename `ASCII-8BIT` encoding to `BINARY`
by shyouhei (Shyouhei Urabe) 24 Jan '24

24 Jan '24

Issue #18576 has been updated by shyouhei (Shyouhei Urabe). @naruse is actually positive for changing error messages (see #note-28). I guess everybody here is agreeing to @byroot's list of proposed changes in #note-37 (except wording)? ---------------------------------------- Feature #18576: Rename `ASCII-8BIT` encoding to `BINARY` https://bugs.ruby-lang.org/issues/18576#change-106416 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal * Target version: 3.4 ---------------------------------------- ### Context I'm now used to it, but something that confused me for years was errors such as: ```ruby >> "fée" + "\xFF".b (irb):3:in `+': incompatible character encodings: UTF-8 and ASCII-8BIT (Encoding::CompatibilityError) ``` When you aren't that familiar with Ruby, it's really not evident that `ASCII-8BIT` basically means "no encoding" or "binary". And even when you know it, if you don't read carefully it's very easily confused with `US-ASCII`. The `Encoding::BINARY` alias is much more telling IMHO. ### Proposal Since `Encoding::ASCII_8BIT` has been aliased as `Encoding::BINARY` for years, I think renaming it to `BINARY` and then making asking `ASCII_8BIT` the alias would significantly improve usability without backward compatibility concerns. The only concern I could see would be the consistency with a handful of C API functions: - `rb_encoding *rb_ascii8bit_encoding(void)` - `int rb_ascii8bit_encindex(void)` - `VALUE rb_io_ascii8bit_binmode(VALUE io)` But that's for much more advanced users, so I don't think it's much of a concern. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:116028] [Ruby master Feature#20152] mkmf / extconf: Add a proper way to not compile the extension
by byroot (Jean Boussier) 24 Jan '24

24 Jan '24

Issue #20152 has been reported by byroot (Jean Boussier). ---------------------------------------- Feature #20152: mkmf / extconf: Add a proper way to not compile the extension https://bugs.ruby-lang.org/issues/20152 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal ---------------------------------------- ### Context There are various gems that ship with a native extension as a way to speedup part of the gem, but also ship with a pure Ruby version of these methods as a fallback. So they only want to compile the extension if the platform supports it, and if not, just fallback to the slightly slower Ruby version. Right now users rely on one of two hacks to do this. Either they create an empty Makefile, but then still depend on `make` being available, or publish platform specific packages without any extension in them. Examples: - [`bootsnap` skip compilation if not on MRI or TruffleRuby](https://github.com/Shopify/bootsnap/blob/070151f1305f23102365d… - [`erb` has an extension for MRI but then need to publish a `java` version of the gem that doesn't actually contain Java code, just to skip compilation on JRuby](https://github.com/ruby/erb/issues/52) - [`hiredis-client` skips the compilation for Windows and non-MRI rubies](https://github.com/redis-rb/redis-client/blob/1ab081c1d0e47df5d55e0… ### Feature It would be very useful to have some proper first class API to skip compiling the extension. Something like: ```ruby require "mkmf" if RUBY_ENGINE != "ruby" || RUBY_PLATFORM.match?(/mswin/) skip_compilation else # ... end ``` cc @k0kubun @headius -- https://bugs.ruby-lang.org/

10 20

[ruby-core:116379] [Ruby master Feature#19057] Hide implementation of `rb_io_t`.
by Eregon (Benoit Daloze) 24 Jan '24

24 Jan '24

Issue #19057 has been updated by Eregon (Benoit Daloze). Agreed, from the hardware POV it has to store the struct in a reasonable/consistent manner in memory. I would be more worried about the C compiler, if the C compiler figures out a pointer is casted to "unrelated" struct types it might consider that as undefined behavior and do anything. But I suspect that's explicitly not undefined behavior, because C has no notion of struct types being related or not, and there is likely tons of software using structs "inheriting/extending" another smaller struct. It might not explicitly be allowed either because I guess the fully correct way to do this is using a union of both structs, and not casting to a specific struct directly then. ---------------------------------------- Feature #19057: Hide implementation of `rb_io_t`. https://bugs.ruby-lang.org/issues/19057#change-106396 * Author: ioquatix (Samuel Williams) * Status: Assigned * Priority: Normal * Assignee: ioquatix (Samuel Williams) * Target version: 3.4 ---------------------------------------- In order to make improvements to the IO implementation like <https://bugs.ruby-lang.org/issues/18455>, we need to add new fields to `struct rb_io_t`. By the way, ending types in `_t` is not recommended by POSIX, so I'm also trying to rename the internal implementation to drop `_t` where possible during this conversion. Anyway, we should try to hide the implementation of `struct rb_io`. Ideally, we don't expose any of it, but the problem is backwards compatibility. So, in order to remain backwards compatibility, we should expose some fields of `struct rb_io`, the most commonly used one is `fd` and `mode`, but several others are commonly used. There are many fields which should not be exposed because they are implementation details. ## Current proposal The current proposed change <https://github.com/ruby/ruby/pull/6511> creates two structs: ```c // include/ruby/io.h #ifndef RB_IO_T struct rb_io { int fd; // ... public fields ... }; #else struct rb_io; #endif // internal/io.h #define RB_IO_T struct rb_io { int fd; // ... public fields ... // ... private fields ... }; ``` However, we are not 100% confident this is safe according to the C specification. My experience is not sufficiently wide to say this is safe in practice, but it does look okay to both myself, and @Eregon + @tenderlovemaking have both given some kind of approval. That being said, maybe it's not safe. There are two alternatives: ## Hide all details We can make public `struct rb_io` completely invisible. ```c // include/ruby/io.h #define RB_IO_HIDDEN struct rb_io; int rb_ioptr_descriptor(struct rb_io *ioptr); // accessor for previously visible state. // internal/io.h struct rb_io { // ... all fields ... }; ``` This would only be forwards compatible, and code would need to feature detect like this: ```c #ifdef RB_IO_HIDDEN #define RB_IOPTR_DESCRIPTOR rb_ioptr_descriptor #else #define RB_IOPTR_DESCRIPTOR(ioptr) rb_ioptr_descriptor(ioptr) #endif ``` ## Nested public interface Alternatively, we can nest the public fields into the private struct: ```c // include/ruby/io.h struct rb_io_public { int fd; // ... public fields ... }; // internal/io.h #define RB_IO_T struct rb_io { struct rb_io_public public; // ... private fields ... }; ``` ## Considerations I personally think the "Hide all details" implementation is the best, but it's also the lest compatible. This is also what we are ultimately aiming for, whether we decide to take an intermediate "compatibility step" is up to us. I think "Nested public interface" is messy and introduces more complexity, but it might be slightly better defined than the "Current proposal" which might create undefined behaviour. That being said, all the tests are passing. -- https://bugs.ruby-lang.org/

2 1

[ruby-core:116388] [BUG] leak in deprecated Data_Make_Struct
by Eric Wong 24 Jan '24

24 Jan '24

I can't open new bug on Redmine due to TOTP so I'll post here. (I'm not going to run JS, view images, or deal with proprietary tools/services). I realize Data_Make_Struct is deprecated and I'm updating a gem to TypedData, there's likely old code which is affected by this leak. With the patch below to make data_spec.c use Data_Make_Struct, this script leaks: $LOAD_PATH << './spec/ruby/optional/capi/ext' require 'data_spec' loop { CApiAllocSpecs.Data_Make_Struct } I noticed this on Ruby 3.3 (32-bit x86) with `clogger' RubyGem, and also up to recent master commit c0cabc0a699b2c8b (Dump annotations on RubyVM::ISeq.disasm (#9667), 2024-01-23) --- Not sure how to write a Ruby spec for malloc memory leaks but I'm just a drive by contributor at this point. It takes me 2-3 hours to build and test Ruby on my old system so I don't feel like doing more. spec/ruby/optional/capi/ext/data_spec.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/spec/ruby/optional/capi/ext/data_spec.c b/spec/ruby/optional/capi/ext/data_spec.c index ef069ef0ba..450eef6c76 100644 --- a/spec/ruby/optional/capi/ext/data_spec.c +++ b/spec/ruby/optional/capi/ext/data_spec.c @@ -24,6 +24,11 @@ VALUE sdaf_alloc_func(VALUE klass) { return Data_Wrap_Struct(klass, &sample_wrapped_struct_mark, &sample_wrapped_struct_free, bar); } +VALUE sdaf_make_struct(VALUE klass) { + struct sample_wrapped_struct* bar; + return Data_Make_Struct(klass, struct sample_wrapped_struct, sample_wrapped_struct_mark, RUBY_DEFAULT_FREE, bar); +} + VALUE sdaf_get_struct(VALUE self) { struct sample_wrapped_struct* bar; Data_Get_Struct(self, struct sample_wrapped_struct, bar); @@ -74,6 +79,7 @@ VALUE sws_rb_check_type(VALUE self, VALUE obj, VALUE other) { void Init_data_spec(void) { VALUE cls = rb_define_class("CApiAllocSpecs", rb_cObject); rb_define_alloc_func(cls, sdaf_alloc_func); + rb_define_singleton_method(cls, "Data_Make_Struct", sdaf_make_struct, 0); rb_define_method(cls, "wrapped_data", sdaf_get_struct, 0); cls = rb_define_class("CApiWrappedStructSpecs", rb_cObject); rb_define_method(cls, "wrap_struct", sws_wrap_struct, 1);

1 0

[ruby-core:116381] [Ruby master Feature#8506] Object#iter_for / Object#to_iter
by Eregon (Benoit Daloze) 23 Jan '24

23 Jan '24

Issue #8506 has been updated by Eregon (Benoit Daloze). Status changed from Open to Rejected Right, so given they do the same or very similar, let's close this. ---------------------------------------- Feature #8506: Object#iter_for / Object#to_iter https://bugs.ruby-lang.org/issues/8506#change-106398 * Author: alindeman (Andy Lindeman) * Status: Rejected * Priority: Normal ---------------------------------------- =begin Ruby's (({Enumerator})) and (({#enum_for})) methods are very powerful and I use them very often. However, (({Object#enum_for})) requires a method that yields, usually in some sort of loop. Many objects in Ruby have methods that iterate to a "next value," but do not yield. For example, (({Fixnum#next.})) There is no way to use (({Fixnum#next})) with (({#enum_for})) directly that I am aware of. I propose the introduction of (({Object#iter_for})) which--given a method--generates a lazy sequence by continually invoking the method on successive values. I call it (({iter})) or (({iterate})) because it is very similar to clojure's iterate: http://clojure.github.io/clojure/clojure.core-api.html#clojure.core/iterate Proposed API: 0.iter_for(:next).take(5) # => [0, 1, 2, 3, 4] require 'date' Date.new(2013, 1, 1).iter_for(:next_month).take(3) # => [Tue, 01 Jan 2013, Fri, 01 Feb 2013, Fri, 01 Mar 2013] I am especially excited about (({0.iter_for(:next)})) as I find myself using infinite lazy numeric sequences more often lately to solve specific kinds of problems. Right now you are required to write something like: (({Enumerator.new { |y| i = 0; loop { y << i; i += 1 } }})) or (({(0..Float::INFINITY).each})). Neither is especially elegant or happy to the developers' eyes in my opinion. Thank you all :) Ruby is an amazing tool. =end -- https://bugs.ruby-lang.org/

1 0

[ruby-core:116380] [Ruby master Feature#8506] Object#iter_for / Object#to_iter
by p8 (Petrik de Heus) 23 Jan '24

23 Jan '24

Issue #8506 has been updated by p8 (Petrik de Heus). ``` 0.iter_for(:next).take(5) # => [0, 1, 2, 3, 4] ```ruby Enumerator.produce(0, &:succ).take(5) => [0, 1, 2, 3, 4] Enumerator.produce(Date.new(2013, 1, 1), &:next_month).take(3) => [Tue, 01 Jan 2013, Fri, 01 Feb 2013, Fri, 01 Mar 2013] ``` ---------------------------------------- Feature #8506: Object#iter_for / Object#to_iter https://bugs.ruby-lang.org/issues/8506#change-106397 * Author: alindeman (Andy Lindeman) * Status: Open * Priority: Normal ---------------------------------------- =begin Ruby's (({Enumerator})) and (({#enum_for})) methods are very powerful and I use them very often. However, (({Object#enum_for})) requires a method that yields, usually in some sort of loop. Many objects in Ruby have methods that iterate to a "next value," but do not yield. For example, (({Fixnum#next.})) There is no way to use (({Fixnum#next})) with (({#enum_for})) directly that I am aware of. I propose the introduction of (({Object#iter_for})) which--given a method--generates a lazy sequence by continually invoking the method on successive values. I call it (({iter})) or (({iterate})) because it is very similar to clojure's iterate: http://clojure.github.io/clojure/clojure.core-api.html#clojure.core/iterate Proposed API: 0.iter_for(:next).take(5) # => [0, 1, 2, 3, 4] require 'date' Date.new(2013, 1, 1).iter_for(:next_month).take(3) # => [Tue, 01 Jan 2013, Fri, 01 Feb 2013, Fri, 01 Mar 2013] I am especially excited about (({0.iter_for(:next)})) as I find myself using infinite lazy numeric sequences more often lately to solve specific kinds of problems. Right now you are required to write something like: (({Enumerator.new { |y| i = 0; loop { y << i; i += 1 } }})) or (({(0..Float::INFINITY).each})). Neither is especially elegant or happy to the developers' eyes in my opinion. Thank you all :) Ruby is an amazing tool. =end -- https://bugs.ruby-lang.org/

1 0

[ruby-core:116213] [Ruby master Feature#19057] Hide implementation of `rb_io_t`.
by ioquatix (Samuel Williams) 23 Jan '24

23 Jan '24

Issue #19057 has been updated by ioquatix (Samuel Williams). Are you saying that in general: ``` struct A { int x; float y; char z; struct S s; }; ``` and ``` struct B { int x; float y; }; ``` That reinterpreting a pointer to A as a pointer to B, and accessing field `B::y` is never undefined? And that generalises? The only rule I knew was the first field, if a struct, is guaranteed to be at offset=0. Are you able to link me to the standard? While I understand what you are saying may be true, I was not able to find confirmation of it in the standard. ---------------------------------------- Feature #19057: Hide implementation of `rb_io_t`. https://bugs.ruby-lang.org/issues/19057#change-106233 * Author: ioquatix (Samuel Williams) * Status: Assigned * Priority: Normal * Assignee: ioquatix (Samuel Williams) * Target version: 3.4 ---------------------------------------- In order to make improvements to the IO implementation like <https://bugs.ruby-lang.org/issues/18455>, we need to add new fields to `struct rb_io_t`. By the way, ending types in `_t` is not recommended by POSIX, so I'm also trying to rename the internal implementation to drop `_t` where possible during this conversion. Anyway, we should try to hide the implementation of `struct rb_io`. Ideally, we don't expose any of it, but the problem is backwards compatibility. So, in order to remain backwards compatibility, we should expose some fields of `struct rb_io`, the most commonly used one is `fd` and `mode`, but several others are commonly used. There are many fields which should not be exposed because they are implementation details. ## Current proposal The current proposed change <https://github.com/ruby/ruby/pull/6511> creates two structs: ```c // include/ruby/io.h #ifndef RB_IO_T struct rb_io { int fd; // ... public fields ... }; #else struct rb_io; #endif // internal/io.h #define RB_IO_T struct rb_io { int fd; // ... public fields ... // ... private fields ... }; ``` However, we are not 100% confident this is safe according to the C specification. My experience is not sufficiently wide to say this is safe in practice, but it does look okay to both myself, and @Eregon + @tenderlovemaking have both given some kind of approval. That being said, maybe it's not safe. There are two alternatives: ## Hide all details We can make public `struct rb_io` completely invisible. ```c // include/ruby/io.h #define RB_IO_HIDDEN struct rb_io; int rb_ioptr_descriptor(struct rb_io *ioptr); // accessor for previously visible state. // internal/io.h struct rb_io { // ... all fields ... }; ``` This would only be forwards compatible, and code would need to feature detect like this: ```c #ifdef RB_IO_HIDDEN #define RB_IOPTR_DESCRIPTOR rb_ioptr_descriptor #else #define RB_IOPTR_DESCRIPTOR(ioptr) rb_ioptr_descriptor(ioptr) #endif ``` ## Nested public interface Alternatively, we can nest the public fields into the private struct: ```c // include/ruby/io.h struct rb_io_public { int fd; // ... public fields ... }; // internal/io.h #define RB_IO_T struct rb_io { struct rb_io_public public; // ... private fields ... }; ``` ## Considerations I personally think the "Hide all details" implementation is the best, but it's also the lest compatible. This is also what we are ultimately aiming for, whether we decide to take an intermediate "compatibility step" is up to us. I think "Nested public interface" is messy and introduces more complexity, but it might be slightly better defined than the "Current proposal" which might create undefined behaviour. That being said, all the tests are passing. -- https://bugs.ruby-lang.org/

2 1

[ruby-core:116368] [Ruby master Misc#20201] Should `opt_reverse` be renamed?
by kddnewton (Kevin Newton) 22 Jan '24

22 Jan '24

Issue #20201 has been reported by kddnewton (Kevin Newton). ---------------------------------------- Misc #20201: Should `opt_reverse` be renamed? https://bugs.ruby-lang.org/issues/20201 * Author: kddnewton (Kevin Newton) * Status: Open * Priority: Normal ---------------------------------------- When you use keywords with the &&= or ||= operator on a [] method call, it uses the `opt_reverse` instruction. For example: ``` ruby h[foo: 1] &&= 2 ``` This instruction is used regardless of if optimizations are on or off. (`--dump=insns_without_opt` and `RubyVM::InstructionSequence.compile_option = false` both don't turn it off.) e`opt_reverse` is also used within the peephole optimizer, so its naming makes sense in those cases. Should the instruction be renamed `reverse`/`stack_reverse`? -- https://bugs.ruby-lang.org/

1 0

[ruby-core:116363] [Ruby master Feature#18576] Rename `ASCII-8BIT` encoding to `BINARY`
by Eregon (Benoit Daloze) 22 Jan '24

22 Jan '24

Issue #18576 has been updated by Eregon (Benoit Daloze). I think for that last example, omitting `ASCII-8BIT` would be much clearer, also two sets of parens seems too much. So: ``` (irb):8:in `+': incompatible character encodings: UTF-8 and BINARY (Encoding::CompatibilityError) ``` Otherwise we would likely still have the confusion that "ASCII" is not compatible with UTF-8 (which is untrue of course). ---------------------------------------- Feature #18576: Rename `ASCII-8BIT` encoding to `BINARY` https://bugs.ruby-lang.org/issues/18576#change-106383 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal * Target version: 3.4 ---------------------------------------- ### Context I'm now used to it, but something that confused me for years was errors such as: ```ruby >> "fée" + "\xFF".b (irb):3:in `+': incompatible character encodings: UTF-8 and ASCII-8BIT (Encoding::CompatibilityError) ``` When you aren't that familiar with Ruby, it's really not evident that `ASCII-8BIT` basically means "no encoding" or "binary". And even when you know it, if you don't read carefully it's very easily confused with `US-ASCII`. The `Encoding::BINARY` alias is much more telling IMHO. ### Proposal Since `Encoding::ASCII_8BIT` has been aliased as `Encoding::BINARY` for years, I think renaming it to `BINARY` and then making asking `ASCII_8BIT` the alias would significantly improve usability without backward compatibility concerns. The only concern I could see would be the consistency with a handful of C API functions: - `rb_encoding *rb_ascii8bit_encoding(void)` - `int rb_ascii8bit_encindex(void)` - `VALUE rb_io_ascii8bit_binmode(VALUE io)` But that's for much more advanced users, so I don't think it's much of a concern. -- https://bugs.ruby-lang.org/

1 0