[ruby-core:118388] [Ruby master Feature#20594] A new String method to append bytes while preserving encoding
Issue #20594 has been reported by byroot (Jean Boussier). ---------------------------------------- Feature #20594: A new String method to append bytes while preserving encoding https://bugs.ruby-lang.org/issues/20594 * Author: byroot (Jean Boussier) * Status: Open ---------------------------------------- ### Context When working with binary protocols such as `protobuf` or `MessagePack`, you may often need to assemble multiple strings of different encoding: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title << 255 << body.bytesize << body end end Post.new("Hello", "World").serialize("somedata".b) # => "somedata\xFF\x05Hello\xFF\x05World" #<Encoding:ASCII-8BIT> ``` The problem in the above case, is that because `Encoding::ASCII_8BIT` is declared as ASCII compatible, if one of the appended string contains bytes outside the ASCII range, string is automatically promoted to another encoding, which then leads to encoding issues: ```ruby Post.new("H€llo", "Wôrld").serialize("somedata".b) # => incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError) ``` In many cases, you want to append to a String without changing the receiver's encoding. The issue isn't exclusive to binary protocols and formats, it also happen with ASCII protocols that accept arbitrary bytes inline, like Redis's RESP protocol or even HTTP/1.1. ### Previous discussion There was a similar feature request a while ago, but it was abandoned: https://bugs.ruby-lang.org/issues/14975 ### Existing solutions You can of course always cast the strings you append to avoid this problem: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title.b << 255 << body.bytesize << body.b end end ``` But this cause a lot of needless allocations. You'd think you could also use `bytesplice`, but it actually has the same issue: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize buf.bytesplice(buf.bytesize, title.bytesize, title) buf << 255 << body.bytesize buf.bytesplice(buf.bytesize, body.bytesize, title) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => 'String#bytesplice': incompatible character encodings: BINARY (ASCII-8BIT) and UTF-8 (Encoding::CompatibilityError) ``` And even if it worked, it would be very unergonomic. ### Proposal: a `byteconcat` method A solution to this would be to add a new `byteconcat` method, that could be shimed as: ```ruby class String def byteconcat(*strings) strings.map! do |s| if s.is_a?(String) && s.encoding != encoding s.dup.force_encoding(encoding) else s end end concat(*strings) end end Post = Struct.new(:title, :body) do def serialize(buf) buf.byteconcat( 255, title.bytesize, title, 255, body.bytesize, body, ) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => "somedata\xFF\aH\xE2\x82\xACllo\xFF\x06W\xC3\xB4rld" #<Encoding:ASCII-8BIT> ``` But of course a builtin implementation wouldn't need to dup the arguments. Like other `byte*` methods, it's the responsibility of the caller to ensure the resulting string has a valid encoding, or to deal with it if not. ### Method name and signature #### Name This proposal suggests `String#byteconcat`, to mirror `String#concat`, but other names are possible: - `byteappend` (like `Array#append`) - `bytepush` (like `Array#push`) #### Signature This proposal makes `byteconcat` accept either `String` or `Integer` (in char range) arguments like `concat`. I believe it makes sense for consistency and also because it's not uncommon for protocols to have some byte based segments, and Integers are more convenient there. The proposed method also accept variable arguments for consistency with `String#concat`, `Array#push`, `Array#append`. The proposed method returns self, like `concat` and others. ### YJIT consideration I consulted @maximecb about this proposal, and according to her, accepting variable arguments makes it harder for YJIT to optimize. I suspect consistency with other APIs trumps the performance consideration, but I think it's worth mentioning. -- https://bugs.ruby-lang.org/
Issue #20594 has been updated by Eregon (Benoit Daloze). +1 from me. I forgot that `String#concat` accepted integers. Together with accepting variable arguments, it feels like this new method would be quite a bit complicated to implement and optimize well. But, it's consistent with `String#concat` so I think it makes sense as proposed. ---------------------------------------- Feature #20594: A new String method to append bytes while preserving encoding https://bugs.ruby-lang.org/issues/20594#change-108905 * Author: byroot (Jean Boussier) * Status: Open ---------------------------------------- ### Context When working with binary protocols such as `protobuf` or `MessagePack`, you may often need to assemble multiple strings of different encoding: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title << 255 << body.bytesize << body end end Post.new("Hello", "World").serialize("somedata".b) # => "somedata\xFF\x05Hello\xFF\x05World" #<Encoding:ASCII-8BIT> ``` The problem in the above case, is that because `Encoding::ASCII_8BIT` is declared as ASCII compatible, if one of the appended string contains bytes outside the ASCII range, string is automatically promoted to another encoding, which then leads to encoding issues: ```ruby Post.new("H€llo", "Wôrld").serialize("somedata".b) # => incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError) ``` In many cases, you want to append to a String without changing the receiver's encoding. The issue isn't exclusive to binary protocols and formats, it also happen with ASCII protocols that accept arbitrary bytes inline, like Redis's RESP protocol or even HTTP/1.1. ### Previous discussion There was a similar feature request a while ago, but it was abandoned: https://bugs.ruby-lang.org/issues/14975 ### Existing solutions You can of course always cast the strings you append to avoid this problem: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title.b << 255 << body.bytesize << body.b end end ``` But this cause a lot of needless allocations. You'd think you could also use `bytesplice`, but it actually has the same issue: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize buf.bytesplice(buf.bytesize, title.bytesize, title) buf << 255 << body.bytesize buf.bytesplice(buf.bytesize, body.bytesize, title) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => 'String#bytesplice': incompatible character encodings: BINARY (ASCII-8BIT) and UTF-8 (Encoding::CompatibilityError) ``` And even if it worked, it would be very unergonomic. ### Proposal: a `byteconcat` method A solution to this would be to add a new `byteconcat` method, that could be shimed as: ```ruby class String def byteconcat(*strings) strings.map! do |s| if s.is_a?(String) && s.encoding != encoding s.dup.force_encoding(encoding) else s end end concat(*strings) end end Post = Struct.new(:title, :body) do def serialize(buf) buf.byteconcat( 255, title.bytesize, title, 255, body.bytesize, body, ) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => "somedata\xFF\aH\xE2\x82\xACllo\xFF\x06W\xC3\xB4rld" #<Encoding:ASCII-8BIT> ``` But of course a builtin implementation wouldn't need to dup the arguments. Like other `byte*` methods, it's the responsibility of the caller to ensure the resulting string has a valid encoding, or to deal with it if not. ### Method name and signature #### Name This proposal suggests `String#byteconcat`, to mirror `String#concat`, but other names are possible: - `byteappend` (like `Array#append`) - `bytepush` (like `Array#push`) #### Signature This proposal makes `byteconcat` accept either `String` or `Integer` (in char range) arguments like `concat`. I believe it makes sense for consistency and also because it's not uncommon for protocols to have some byte based segments, and Integers are more convenient there. The proposed method also accept variable arguments for consistency with `String#concat`, `Array#push`, `Array#append`. The proposed method returns self, like `concat` and others. ### YJIT consideration I consulted @maximecb about this proposal, and according to her, accepting variable arguments makes it harder for YJIT to optimize. I suspect consistency with other APIs trumps the performance consideration, but I think it's worth mentioning. -- https://bugs.ruby-lang.org/
Issue #20594 has been updated by maximecb (Maxime Chevalier-Boisvert).
I consulted @maximecb (Maxime Chevalier-Boisvert) about this proposal, and according to her, accepting variable arguments makes it harder for YJIT to optimize.
Yeah my personal preference would be for something like `bytepush` or `byteappend` that accepts a single argument, the simplest possible method, that is kept as strict as possible, while still getting the job done. The technical reason being that in order to handle multiple arguments, we have to essentially unroll the loop in the YJIT codegen, and speculate that the arguments will all keep the same type at run-time. This is likely to be true, but it means that we have to speculate on many things at once and then generate a big piece of machine code all at the same time. I understand that there is some temptation to keep the API more similar to other String methods with a similar purpose, but IMO this method is already unusual because it's essentially an `unsafe` operation (no encoding validation). The reason we're discussing this method in the first place is performance, and so maybe performance/simplicity should be the main concern. If the only way this change will be allowed to happen is to allow a variable argument count, then OK I guess, but I would like to push for something more simple, less dynamic. We sort of have the benefit of hindsight here, but take the Ruby binding API for example. If we were designing Ruby today, there is no way we would choose to make binding as powerful and unrestricted as it is. It's much easier to remove restrictions later than to add them back in. ---------------------------------------- Feature #20594: A new String method to append bytes while preserving encoding https://bugs.ruby-lang.org/issues/20594#change-108906 * Author: byroot (Jean Boussier) * Status: Open ---------------------------------------- ### Context When working with binary protocols such as `protobuf` or `MessagePack`, you may often need to assemble multiple strings of different encoding: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title << 255 << body.bytesize << body end end Post.new("Hello", "World").serialize("somedata".b) # => "somedata\xFF\x05Hello\xFF\x05World" #<Encoding:ASCII-8BIT> ``` The problem in the above case, is that because `Encoding::ASCII_8BIT` is declared as ASCII compatible, if one of the appended string contains bytes outside the ASCII range, string is automatically promoted to another encoding, which then leads to encoding issues: ```ruby Post.new("H€llo", "Wôrld").serialize("somedata".b) # => incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError) ``` In many cases, you want to append to a String without changing the receiver's encoding. The issue isn't exclusive to binary protocols and formats, it also happen with ASCII protocols that accept arbitrary bytes inline, like Redis's RESP protocol or even HTTP/1.1. ### Previous discussion There was a similar feature request a while ago, but it was abandoned: https://bugs.ruby-lang.org/issues/14975 ### Existing solutions You can of course always cast the strings you append to avoid this problem: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title.b << 255 << body.bytesize << body.b end end ``` But this cause a lot of needless allocations. You'd think you could also use `bytesplice`, but it actually has the same issue: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize buf.bytesplice(buf.bytesize, title.bytesize, title) buf << 255 << body.bytesize buf.bytesplice(buf.bytesize, body.bytesize, title) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => 'String#bytesplice': incompatible character encodings: BINARY (ASCII-8BIT) and UTF-8 (Encoding::CompatibilityError) ``` And even if it worked, it would be very unergonomic. ### Proposal: a `byteconcat` method A solution to this would be to add a new `byteconcat` method, that could be shimed as: ```ruby class String def byteconcat(*strings) strings.map! do |s| if s.is_a?(String) && s.encoding != encoding s.dup.force_encoding(encoding) else s end end concat(*strings) end end Post = Struct.new(:title, :body) do def serialize(buf) buf.byteconcat( 255, title.bytesize, title, 255, body.bytesize, body, ) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => "somedata\xFF\aH\xE2\x82\xACllo\xFF\x06W\xC3\xB4rld" #<Encoding:ASCII-8BIT> ``` But of course a builtin implementation wouldn't need to dup the arguments. Like other `byte*` methods, it's the responsibility of the caller to ensure the resulting string has a valid encoding, or to deal with it if not. ### Method name and signature #### Name This proposal suggests `String#byteconcat`, to mirror `String#concat`, but other names are possible: - `byteappend` (like `Array#append`) - `bytepush` (like `Array#push`) #### Signature This proposal makes `byteconcat` accept either `String` or `Integer` (in char range) arguments like `concat`. I believe it makes sense for consistency and also because it's not uncommon for protocols to have some byte based segments, and Integers are more convenient there. The proposed method also accept variable arguments for consistency with `String#concat`, `Array#push`, `Array#append`. The proposed method returns self, like `concat` and others. ### YJIT consideration I consulted @maximecb about this proposal, and according to her, accepting variable arguments makes it harder for YJIT to optimize. I suspect consistency with other APIs trumps the performance consideration, but I think it's worth mentioning. -- https://bugs.ruby-lang.org/
Issue #20594 has been updated by matz (Yukihiro Matsumoto). Assignee set to byroot (Jean Boussier) I agree with having such a method. However, I disagree with the name `byteconcat`. Since other methods with byte prefixes have the behavior of counting the index in bytes, but this method has nothing to do with the index, but with the encoding. If this method only works with `BINARY` encoding, the name might be `binary_concat` or `binconcat`. In the developers' meeting, someone proposed `force_concat`. Matz. ---------------------------------------- Feature #20594: A new String method to append bytes while preserving encoding https://bugs.ruby-lang.org/issues/20594#change-109061 * Author: byroot (Jean Boussier) * Status: Open * Assignee: byroot (Jean Boussier) ---------------------------------------- ### Context When working with binary protocols such as `protobuf` or `MessagePack`, you may often need to assemble multiple strings of different encoding: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title << 255 << body.bytesize << body end end Post.new("Hello", "World").serialize("somedata".b) # => "somedata\xFF\x05Hello\xFF\x05World" #<Encoding:ASCII-8BIT> ``` The problem in the above case, is that because `Encoding::ASCII_8BIT` is declared as ASCII compatible, if one of the appended string contains bytes outside the ASCII range, string is automatically promoted to another encoding, which then leads to encoding issues: ```ruby Post.new("H€llo", "Wôrld").serialize("somedata".b) # => incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError) ``` In many cases, you want to append to a String without changing the receiver's encoding. The issue isn't exclusive to binary protocols and formats, it also happen with ASCII protocols that accept arbitrary bytes inline, like Redis's RESP protocol or even HTTP/1.1. ### Previous discussion There was a similar feature request a while ago, but it was abandoned: https://bugs.ruby-lang.org/issues/14975 ### Existing solutions You can of course always cast the strings you append to avoid this problem: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title.b << 255 << body.bytesize << body.b end end ``` But this cause a lot of needless allocations. You'd think you could also use `bytesplice`, but it actually has the same issue: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize buf.bytesplice(buf.bytesize, title.bytesize, title) buf << 255 << body.bytesize buf.bytesplice(buf.bytesize, body.bytesize, title) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => 'String#bytesplice': incompatible character encodings: BINARY (ASCII-8BIT) and UTF-8 (Encoding::CompatibilityError) ``` And even if it worked, it would be very unergonomic. ### Proposal: a `byteconcat` method A solution to this would be to add a new `byteconcat` method, that could be shimed as: ```ruby class String def byteconcat(*strings) strings.map! do |s| if s.is_a?(String) && s.encoding != encoding s.dup.force_encoding(encoding) else s end end concat(*strings) end end Post = Struct.new(:title, :body) do def serialize(buf) buf.byteconcat( 255, title.bytesize, title, 255, body.bytesize, body, ) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => "somedata\xFF\aH\xE2\x82\xACllo\xFF\x06W\xC3\xB4rld" #<Encoding:ASCII-8BIT> ``` But of course a builtin implementation wouldn't need to dup the arguments. Like other `byte*` methods, it's the responsibility of the caller to ensure the resulting string has a valid encoding, or to deal with it if not. ### Method name and signature #### Name This proposal suggests `String#byteconcat`, to mirror `String#concat`, but other names are possible: - `byteappend` (like `Array#append`) - `bytepush` (like `Array#push`) #### Signature This proposal makes `byteconcat` accept either `String` or `Integer` (in char range) arguments like `concat`. I believe it makes sense for consistency and also because it's not uncommon for protocols to have some byte based segments, and Integers are more convenient there. The proposed method also accept variable arguments for consistency with `String#concat`, `Array#push`, `Array#append`. The proposed method returns self, like `concat` and others. ### YJIT consideration I consulted @maximecb about this proposal, and according to her, accepting variable arguments makes it harder for YJIT to optimize. I suspect consistency with other APIs trumps the performance consideration, but I think it's worth mentioning. -- https://bugs.ruby-lang.org/
Issue #20594 has been updated by byroot (Jean Boussier).
Since other methods with byte prefixes have the behavior of counting the index in bytes, but this method has nothing to do with the index
I kinda disagree here. My understanding is that the `byte*` methods operate on bytes rather than characters, so `size != bytesize`. Here the idea is to concat bytes, not characters so I think it fits.
If this method only works with BINARY encoding,
It doesn't no. It can work for any encoding, you can use it for instance to assemble an UTF-8 string from some network reads: ```ruby buf = +"" # UTF-8 while chunk = io.read(1024) # ASCII-BIT buf.byteconcat(chunk) buf.valid_encoding? # May be true or false end buf ``` ---------------------------------------- Feature #20594: A new String method to append bytes while preserving encoding https://bugs.ruby-lang.org/issues/20594#change-109062 * Author: byroot (Jean Boussier) * Status: Open * Assignee: byroot (Jean Boussier) ---------------------------------------- ### Context When working with binary protocols such as `protobuf` or `MessagePack`, you may often need to assemble multiple strings of different encoding: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title << 255 << body.bytesize << body end end Post.new("Hello", "World").serialize("somedata".b) # => "somedata\xFF\x05Hello\xFF\x05World" #<Encoding:ASCII-8BIT> ``` The problem in the above case, is that because `Encoding::ASCII_8BIT` is declared as ASCII compatible, if one of the appended string contains bytes outside the ASCII range, string is automatically promoted to another encoding, which then leads to encoding issues: ```ruby Post.new("H€llo", "Wôrld").serialize("somedata".b) # => incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError) ``` In many cases, you want to append to a String without changing the receiver's encoding. The issue isn't exclusive to binary protocols and formats, it also happen with ASCII protocols that accept arbitrary bytes inline, like Redis's RESP protocol or even HTTP/1.1. ### Previous discussion There was a similar feature request a while ago, but it was abandoned: https://bugs.ruby-lang.org/issues/14975 ### Existing solutions You can of course always cast the strings you append to avoid this problem: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title.b << 255 << body.bytesize << body.b end end ``` But this cause a lot of needless allocations. You'd think you could also use `bytesplice`, but it actually has the same issue: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize buf.bytesplice(buf.bytesize, title.bytesize, title) buf << 255 << body.bytesize buf.bytesplice(buf.bytesize, body.bytesize, title) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => 'String#bytesplice': incompatible character encodings: BINARY (ASCII-8BIT) and UTF-8 (Encoding::CompatibilityError) ``` And even if it worked, it would be very unergonomic. ### Proposal: a `byteconcat` method A solution to this would be to add a new `byteconcat` method, that could be shimed as: ```ruby class String def byteconcat(*strings) strings.map! do |s| if s.is_a?(String) && s.encoding != encoding s.dup.force_encoding(encoding) else s end end concat(*strings) end end Post = Struct.new(:title, :body) do def serialize(buf) buf.byteconcat( 255, title.bytesize, title, 255, body.bytesize, body, ) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => "somedata\xFF\aH\xE2\x82\xACllo\xFF\x06W\xC3\xB4rld" #<Encoding:ASCII-8BIT> ``` But of course a builtin implementation wouldn't need to dup the arguments. Like other `byte*` methods, it's the responsibility of the caller to ensure the resulting string has a valid encoding, or to deal with it if not. ### Method name and signature #### Name This proposal suggests `String#byteconcat`, to mirror `String#concat`, but other names are possible: - `byteappend` (like `Array#append`) - `bytepush` (like `Array#push`) #### Signature This proposal makes `byteconcat` accept either `String` or `Integer` (in char range) arguments like `concat`. I believe it makes sense for consistency and also because it's not uncommon for protocols to have some byte based segments, and Integers are more convenient there. The proposed method also accept variable arguments for consistency with `String#concat`, `Array#push`, `Array#append`. The proposed method returns self, like `concat` and others. ### YJIT consideration I consulted @maximecb about this proposal, and according to her, accepting variable arguments makes it harder for YJIT to optimize. I suspect consistency with other APIs trumps the performance consideration, but I think it's worth mentioning. -- https://bugs.ruby-lang.org/
Issue #20594 has been updated by byroot (Jean Boussier). Discussed in person in the meeting. I'll think about other names and propose some. ---------------------------------------- Feature #20594: A new String method to append bytes while preserving encoding https://bugs.ruby-lang.org/issues/20594#change-109067 * Author: byroot (Jean Boussier) * Status: Open * Assignee: byroot (Jean Boussier) ---------------------------------------- ### Context When working with binary protocols such as `protobuf` or `MessagePack`, you may often need to assemble multiple strings of different encoding: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title << 255 << body.bytesize << body end end Post.new("Hello", "World").serialize("somedata".b) # => "somedata\xFF\x05Hello\xFF\x05World" #<Encoding:ASCII-8BIT> ``` The problem in the above case, is that because `Encoding::ASCII_8BIT` is declared as ASCII compatible, if one of the appended string contains bytes outside the ASCII range, string is automatically promoted to another encoding, which then leads to encoding issues: ```ruby Post.new("H€llo", "Wôrld").serialize("somedata".b) # => incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError) ``` In many cases, you want to append to a String without changing the receiver's encoding. The issue isn't exclusive to binary protocols and formats, it also happen with ASCII protocols that accept arbitrary bytes inline, like Redis's RESP protocol or even HTTP/1.1. ### Previous discussion There was a similar feature request a while ago, but it was abandoned: https://bugs.ruby-lang.org/issues/14975 ### Existing solutions You can of course always cast the strings you append to avoid this problem: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title.b << 255 << body.bytesize << body.b end end ``` But this cause a lot of needless allocations. You'd think you could also use `bytesplice`, but it actually has the same issue: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize buf.bytesplice(buf.bytesize, title.bytesize, title) buf << 255 << body.bytesize buf.bytesplice(buf.bytesize, body.bytesize, title) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => 'String#bytesplice': incompatible character encodings: BINARY (ASCII-8BIT) and UTF-8 (Encoding::CompatibilityError) ``` And even if it worked, it would be very unergonomic. ### Proposal: a `byteconcat` method A solution to this would be to add a new `byteconcat` method, that could be shimed as: ```ruby class String def byteconcat(*strings) strings.map! do |s| if s.is_a?(String) && s.encoding != encoding s.dup.force_encoding(encoding) else s end end concat(*strings) end end Post = Struct.new(:title, :body) do def serialize(buf) buf.byteconcat( 255, title.bytesize, title, 255, body.bytesize, body, ) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => "somedata\xFF\aH\xE2\x82\xACllo\xFF\x06W\xC3\xB4rld" #<Encoding:ASCII-8BIT> ``` But of course a builtin implementation wouldn't need to dup the arguments. Like other `byte*` methods, it's the responsibility of the caller to ensure the resulting string has a valid encoding, or to deal with it if not. ### Method name and signature #### Name This proposal suggests `String#byteconcat`, to mirror `String#concat`, but other names are possible: - `byteappend` (like `Array#append`) - `bytepush` (like `Array#push`) #### Signature This proposal makes `byteconcat` accept either `String` or `Integer` (in char range) arguments like `concat`. I believe it makes sense for consistency and also because it's not uncommon for protocols to have some byte based segments, and Integers are more convenient there. The proposed method also accept variable arguments for consistency with `String#concat`, `Array#push`, `Array#append`. The proposed method returns self, like `concat` and others. ### YJIT consideration I consulted @maximecb about this proposal, and according to her, accepting variable arguments makes it harder for YJIT to optimize. I suspect consistency with other APIs trumps the performance consideration, but I think it's worth mentioning. -- https://bugs.ruby-lang.org/
Issue #20594 has been updated by mame (Yusuke Endoh). Existing methods with byte-prefix (String#byteindex, #bytesplite, etc.) mean that the unit of offset or size is in byte. `byteconcat` and `byteappend`, on the other hand, are confusing because they have no offset or size, but they mean concatenation without regard to encodings. There were some alternative proposals that came out of the dev meeting. * `force_concat` (matz: it lacks "encoding", I don't like it so much) * `binary_concat` (It should work only when the receiver's encoding is BINARY. Does it fit with @byroot 's motivation?) ---------------------------------------- Feature #20594: A new String method to append bytes while preserving encoding https://bugs.ruby-lang.org/issues/20594#change-109075 * Author: byroot (Jean Boussier) * Status: Open * Assignee: byroot (Jean Boussier) ---------------------------------------- ### Context When working with binary protocols such as `protobuf` or `MessagePack`, you may often need to assemble multiple strings of different encoding: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title << 255 << body.bytesize << body end end Post.new("Hello", "World").serialize("somedata".b) # => "somedata\xFF\x05Hello\xFF\x05World" #<Encoding:ASCII-8BIT> ``` The problem in the above case, is that because `Encoding::ASCII_8BIT` is declared as ASCII compatible, if one of the appended string contains bytes outside the ASCII range, string is automatically promoted to another encoding, which then leads to encoding issues: ```ruby Post.new("H€llo", "Wôrld").serialize("somedata".b) # => incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError) ``` In many cases, you want to append to a String without changing the receiver's encoding. The issue isn't exclusive to binary protocols and formats, it also happen with ASCII protocols that accept arbitrary bytes inline, like Redis's RESP protocol or even HTTP/1.1. ### Previous discussion There was a similar feature request a while ago, but it was abandoned: https://bugs.ruby-lang.org/issues/14975 ### Existing solutions You can of course always cast the strings you append to avoid this problem: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title.b << 255 << body.bytesize << body.b end end ``` But this cause a lot of needless allocations. You'd think you could also use `bytesplice`, but it actually has the same issue: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize buf.bytesplice(buf.bytesize, title.bytesize, title) buf << 255 << body.bytesize buf.bytesplice(buf.bytesize, body.bytesize, title) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => 'String#bytesplice': incompatible character encodings: BINARY (ASCII-8BIT) and UTF-8 (Encoding::CompatibilityError) ``` And even if it worked, it would be very unergonomic. ### Proposal: a `byteconcat` method A solution to this would be to add a new `byteconcat` method, that could be shimed as: ```ruby class String def byteconcat(*strings) strings.map! do |s| if s.is_a?(String) && s.encoding != encoding s.dup.force_encoding(encoding) else s end end concat(*strings) end end Post = Struct.new(:title, :body) do def serialize(buf) buf.byteconcat( 255, title.bytesize, title, 255, body.bytesize, body, ) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => "somedata\xFF\aH\xE2\x82\xACllo\xFF\x06W\xC3\xB4rld" #<Encoding:ASCII-8BIT> ``` But of course a builtin implementation wouldn't need to dup the arguments. Like other `byte*` methods, it's the responsibility of the caller to ensure the resulting string has a valid encoding, or to deal with it if not. ### Method name and signature #### Name This proposal suggests `String#byteconcat`, to mirror `String#concat`, but other names are possible: - `byteappend` (like `Array#append`) - `bytepush` (like `Array#push`) #### Signature This proposal makes `byteconcat` accept either `String` or `Integer` (in char range) arguments like `concat`. I believe it makes sense for consistency and also because it's not uncommon for protocols to have some byte based segments, and Integers are more convenient there. The proposed method also accept variable arguments for consistency with `String#concat`, `Array#push`, `Array#append`. The proposed method returns self, like `concat` and others. ### YJIT consideration I consulted @maximecb about this proposal, and according to her, accepting variable arguments makes it harder for YJIT to optimize. I suspect consistency with other APIs trumps the performance consideration, but I think it's worth mentioning. -- https://bugs.ruby-lang.org/
Issue #20594 has been updated by byroot (Jean Boussier).
binary_concat (It should work only when the receiver's encoding is BINARY. Does it fit with @byroot (Jean Boussier) 's motivation?)
No it's too limiting. It should work with any encoding in my opinion. I'll try to come up with other names. So far I'm thinking of `String#append_bytes(String)`. ---------------------------------------- Feature #20594: A new String method to append bytes while preserving encoding https://bugs.ruby-lang.org/issues/20594#change-109076 * Author: byroot (Jean Boussier) * Status: Open * Assignee: byroot (Jean Boussier) ---------------------------------------- ### Context When working with binary protocols such as `protobuf` or `MessagePack`, you may often need to assemble multiple strings of different encoding: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title << 255 << body.bytesize << body end end Post.new("Hello", "World").serialize("somedata".b) # => "somedata\xFF\x05Hello\xFF\x05World" #<Encoding:ASCII-8BIT> ``` The problem in the above case, is that because `Encoding::ASCII_8BIT` is declared as ASCII compatible, if one of the appended string contains bytes outside the ASCII range, string is automatically promoted to another encoding, which then leads to encoding issues: ```ruby Post.new("H€llo", "Wôrld").serialize("somedata".b) # => incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError) ``` In many cases, you want to append to a String without changing the receiver's encoding. The issue isn't exclusive to binary protocols and formats, it also happen with ASCII protocols that accept arbitrary bytes inline, like Redis's RESP protocol or even HTTP/1.1. ### Previous discussion There was a similar feature request a while ago, but it was abandoned: https://bugs.ruby-lang.org/issues/14975 ### Existing solutions You can of course always cast the strings you append to avoid this problem: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title.b << 255 << body.bytesize << body.b end end ``` But this cause a lot of needless allocations. You'd think you could also use `bytesplice`, but it actually has the same issue: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize buf.bytesplice(buf.bytesize, title.bytesize, title) buf << 255 << body.bytesize buf.bytesplice(buf.bytesize, body.bytesize, title) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => 'String#bytesplice': incompatible character encodings: BINARY (ASCII-8BIT) and UTF-8 (Encoding::CompatibilityError) ``` And even if it worked, it would be very unergonomic. ### Proposal: a `byteconcat` method A solution to this would be to add a new `byteconcat` method, that could be shimed as: ```ruby class String def byteconcat(*strings) strings.map! do |s| if s.is_a?(String) && s.encoding != encoding s.dup.force_encoding(encoding) else s end end concat(*strings) end end Post = Struct.new(:title, :body) do def serialize(buf) buf.byteconcat( 255, title.bytesize, title, 255, body.bytesize, body, ) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => "somedata\xFF\aH\xE2\x82\xACllo\xFF\x06W\xC3\xB4rld" #<Encoding:ASCII-8BIT> ``` But of course a builtin implementation wouldn't need to dup the arguments. Like other `byte*` methods, it's the responsibility of the caller to ensure the resulting string has a valid encoding, or to deal with it if not. ### Method name and signature #### Name This proposal suggests `String#byteconcat`, to mirror `String#concat`, but other names are possible: - `byteappend` (like `Array#append`) - `bytepush` (like `Array#push`) #### Signature This proposal makes `byteconcat` accept either `String` or `Integer` (in char range) arguments like `concat`. I believe it makes sense for consistency and also because it's not uncommon for protocols to have some byte based segments, and Integers are more convenient there. The proposed method also accept variable arguments for consistency with `String#concat`, `Array#push`, `Array#append`. The proposed method returns self, like `concat` and others. ### YJIT consideration I consulted @maximecb about this proposal, and according to her, accepting variable arguments makes it harder for YJIT to optimize. I suspect consistency with other APIs trumps the performance consideration, but I think it's worth mentioning. -- https://bugs.ruby-lang.org/
Issue #20594 has been updated by Eregon (Benoit Daloze). mame (Yusuke Endoh) wrote in #note-7:
Existing methods with byte-prefix (String#byteindex, #bytesplite, etc.) mean that the unit of offset or size is in byte.
My understanding of `byte*` methods is they treat the String as a byte array, which implies indices are just byte indices but also that the encoding is ignored (it seems clear when one does `"é".getbyte(0)`). It's (almost) as-if the string had the BINARY encoding for the duration of the operation, but without the overhead to switch to BINARY and back (which notably could cause some extra code range computation, etc). BTW, I would consider `each_byte` also a `byte*` method, and that one does not accept or pass byte indices. So I think it would make sense to extend the meaning of `byte*` methods to be a little more general, just like I explained above. I don't think it was documented to be only about byte indices either. That said, I think `String#append_bytes(String)` sounds fine too. ---------------------------------------- Feature #20594: A new String method to append bytes while preserving encoding https://bugs.ruby-lang.org/issues/20594#change-109083 * Author: byroot (Jean Boussier) * Status: Open * Assignee: byroot (Jean Boussier) ---------------------------------------- ### Context When working with binary protocols such as `protobuf` or `MessagePack`, you may often need to assemble multiple strings of different encoding: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title << 255 << body.bytesize << body end end Post.new("Hello", "World").serialize("somedata".b) # => "somedata\xFF\x05Hello\xFF\x05World" #<Encoding:ASCII-8BIT> ``` The problem in the above case, is that because `Encoding::ASCII_8BIT` is declared as ASCII compatible, if one of the appended string contains bytes outside the ASCII range, string is automatically promoted to another encoding, which then leads to encoding issues: ```ruby Post.new("H€llo", "Wôrld").serialize("somedata".b) # => incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError) ``` In many cases, you want to append to a String without changing the receiver's encoding. The issue isn't exclusive to binary protocols and formats, it also happen with ASCII protocols that accept arbitrary bytes inline, like Redis's RESP protocol or even HTTP/1.1. ### Previous discussion There was a similar feature request a while ago, but it was abandoned: https://bugs.ruby-lang.org/issues/14975 ### Existing solutions You can of course always cast the strings you append to avoid this problem: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title.b << 255 << body.bytesize << body.b end end ``` But this cause a lot of needless allocations. You'd think you could also use `bytesplice`, but it actually has the same issue: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize buf.bytesplice(buf.bytesize, title.bytesize, title) buf << 255 << body.bytesize buf.bytesplice(buf.bytesize, body.bytesize, title) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => 'String#bytesplice': incompatible character encodings: BINARY (ASCII-8BIT) and UTF-8 (Encoding::CompatibilityError) ``` And even if it worked, it would be very unergonomic. ### Proposal: a `byteconcat` method A solution to this would be to add a new `byteconcat` method, that could be shimed as: ```ruby class String def byteconcat(*strings) strings.map! do |s| if s.is_a?(String) && s.encoding != encoding s.dup.force_encoding(encoding) else s end end concat(*strings) end end Post = Struct.new(:title, :body) do def serialize(buf) buf.byteconcat( 255, title.bytesize, title, 255, body.bytesize, body, ) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => "somedata\xFF\aH\xE2\x82\xACllo\xFF\x06W\xC3\xB4rld" #<Encoding:ASCII-8BIT> ``` But of course a builtin implementation wouldn't need to dup the arguments. Like other `byte*` methods, it's the responsibility of the caller to ensure the resulting string has a valid encoding, or to deal with it if not. ### Method name and signature #### Name This proposal suggests `String#byteconcat`, to mirror `String#concat`, but other names are possible: - `byteappend` (like `Array#append`) - `bytepush` (like `Array#push`) #### Signature This proposal makes `byteconcat` accept either `String` or `Integer` (in char range) arguments like `concat`. I believe it makes sense for consistency and also because it's not uncommon for protocols to have some byte based segments, and Integers are more convenient there. The proposed method also accept variable arguments for consistency with `String#concat`, `Array#push`, `Array#append`. The proposed method returns self, like `concat` and others. ### YJIT consideration I consulted @maximecb about this proposal, and according to her, accepting variable arguments makes it harder for YJIT to optimize. I suspect consistency with other APIs trumps the performance consideration, but I think it's worth mentioning. -- https://bugs.ruby-lang.org/
Issue #20594 has been updated by byroot (Jean Boussier).
but also that the encoding is ignored
That was my understanding as well, but given that `bytesplice` does potentially raise `EncodingError`, there is a mismatch here. In this case I very much want to allow breaking a string encoding. ---------------------------------------- Feature #20594: A new String method to append bytes while preserving encoding https://bugs.ruby-lang.org/issues/20594#change-109084 * Author: byroot (Jean Boussier) * Status: Open * Assignee: byroot (Jean Boussier) ---------------------------------------- ### Context When working with binary protocols such as `protobuf` or `MessagePack`, you may often need to assemble multiple strings of different encoding: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title << 255 << body.bytesize << body end end Post.new("Hello", "World").serialize("somedata".b) # => "somedata\xFF\x05Hello\xFF\x05World" #<Encoding:ASCII-8BIT> ``` The problem in the above case, is that because `Encoding::ASCII_8BIT` is declared as ASCII compatible, if one of the appended string contains bytes outside the ASCII range, string is automatically promoted to another encoding, which then leads to encoding issues: ```ruby Post.new("H€llo", "Wôrld").serialize("somedata".b) # => incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError) ``` In many cases, you want to append to a String without changing the receiver's encoding. The issue isn't exclusive to binary protocols and formats, it also happen with ASCII protocols that accept arbitrary bytes inline, like Redis's RESP protocol or even HTTP/1.1. ### Previous discussion There was a similar feature request a while ago, but it was abandoned: https://bugs.ruby-lang.org/issues/14975 ### Existing solutions You can of course always cast the strings you append to avoid this problem: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title.b << 255 << body.bytesize << body.b end end ``` But this cause a lot of needless allocations. You'd think you could also use `bytesplice`, but it actually has the same issue: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize buf.bytesplice(buf.bytesize, title.bytesize, title) buf << 255 << body.bytesize buf.bytesplice(buf.bytesize, body.bytesize, title) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => 'String#bytesplice': incompatible character encodings: BINARY (ASCII-8BIT) and UTF-8 (Encoding::CompatibilityError) ``` And even if it worked, it would be very unergonomic. ### Proposal: a `byteconcat` method A solution to this would be to add a new `byteconcat` method, that could be shimed as: ```ruby class String def byteconcat(*strings) strings.map! do |s| if s.is_a?(String) && s.encoding != encoding s.dup.force_encoding(encoding) else s end end concat(*strings) end end Post = Struct.new(:title, :body) do def serialize(buf) buf.byteconcat( 255, title.bytesize, title, 255, body.bytesize, body, ) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => "somedata\xFF\aH\xE2\x82\xACllo\xFF\x06W\xC3\xB4rld" #<Encoding:ASCII-8BIT> ``` But of course a builtin implementation wouldn't need to dup the arguments. Like other `byte*` methods, it's the responsibility of the caller to ensure the resulting string has a valid encoding, or to deal with it if not. ### Method name and signature #### Name This proposal suggests `String#byteconcat`, to mirror `String#concat`, but other names are possible: - `byteappend` (like `Array#append`) - `bytepush` (like `Array#push`) #### Signature This proposal makes `byteconcat` accept either `String` or `Integer` (in char range) arguments like `concat`. I believe it makes sense for consistency and also because it's not uncommon for protocols to have some byte based segments, and Integers are more convenient there. The proposed method also accept variable arguments for consistency with `String#concat`, `Array#push`, `Array#append`. The proposed method returns self, like `concat` and others. ### YJIT consideration I consulted @maximecb about this proposal, and according to her, accepting variable arguments makes it harder for YJIT to optimize. I suspect consistency with other APIs trumps the performance consideration, but I think it's worth mentioning. -- https://bugs.ruby-lang.org/
Issue #20594 has been updated by Eregon (Benoit Daloze). I wonder if that's a bug of `bytesplice`. It's not like e.g. `setbyte` would respect the encoding (it cannot possibly). OTOH `byteindex` also takes a String argument and raises `Encoding::CompatibilityError` for `"é".b.byteindex "é"` so I guess it's not so clear-cut currently (I get the replies above a bit better). Another way to see the new method is concatenation without encoding negotiation/checking encoding compatibility. Having byte/bytes in the name seems a good way to express that. ---------------------------------------- Feature #20594: A new String method to append bytes while preserving encoding https://bugs.ruby-lang.org/issues/20594#change-109086 * Author: byroot (Jean Boussier) * Status: Open * Assignee: byroot (Jean Boussier) ---------------------------------------- ### Context When working with binary protocols such as `protobuf` or `MessagePack`, you may often need to assemble multiple strings of different encoding: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title << 255 << body.bytesize << body end end Post.new("Hello", "World").serialize("somedata".b) # => "somedata\xFF\x05Hello\xFF\x05World" #<Encoding:ASCII-8BIT> ``` The problem in the above case, is that because `Encoding::ASCII_8BIT` is declared as ASCII compatible, if one of the appended string contains bytes outside the ASCII range, string is automatically promoted to another encoding, which then leads to encoding issues: ```ruby Post.new("H€llo", "Wôrld").serialize("somedata".b) # => incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError) ``` In many cases, you want to append to a String without changing the receiver's encoding. The issue isn't exclusive to binary protocols and formats, it also happen with ASCII protocols that accept arbitrary bytes inline, like Redis's RESP protocol or even HTTP/1.1. ### Previous discussion There was a similar feature request a while ago, but it was abandoned: https://bugs.ruby-lang.org/issues/14975 ### Existing solutions You can of course always cast the strings you append to avoid this problem: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title.b << 255 << body.bytesize << body.b end end ``` But this cause a lot of needless allocations. You'd think you could also use `bytesplice`, but it actually has the same issue: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize buf.bytesplice(buf.bytesize, title.bytesize, title) buf << 255 << body.bytesize buf.bytesplice(buf.bytesize, body.bytesize, title) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => 'String#bytesplice': incompatible character encodings: BINARY (ASCII-8BIT) and UTF-8 (Encoding::CompatibilityError) ``` And even if it worked, it would be very unergonomic. ### Proposal: a `byteconcat` method A solution to this would be to add a new `byteconcat` method, that could be shimed as: ```ruby class String def byteconcat(*strings) strings.map! do |s| if s.is_a?(String) && s.encoding != encoding s.dup.force_encoding(encoding) else s end end concat(*strings) end end Post = Struct.new(:title, :body) do def serialize(buf) buf.byteconcat( 255, title.bytesize, title, 255, body.bytesize, body, ) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => "somedata\xFF\aH\xE2\x82\xACllo\xFF\x06W\xC3\xB4rld" #<Encoding:ASCII-8BIT> ``` But of course a builtin implementation wouldn't need to dup the arguments. Like other `byte*` methods, it's the responsibility of the caller to ensure the resulting string has a valid encoding, or to deal with it if not. ### Method name and signature #### Name This proposal suggests `String#byteconcat`, to mirror `String#concat`, but other names are possible: - `byteappend` (like `Array#append`) - `bytepush` (like `Array#push`) #### Signature This proposal makes `byteconcat` accept either `String` or `Integer` (in char range) arguments like `concat`. I believe it makes sense for consistency and also because it's not uncommon for protocols to have some byte based segments, and Integers are more convenient there. The proposed method also accept variable arguments for consistency with `String#concat`, `Array#push`, `Array#append`. The proposed method returns self, like `concat` and others. ### YJIT consideration I consulted @maximecb about this proposal, and according to her, accepting variable arguments makes it harder for YJIT to optimize. I suspect consistency with other APIs trumps the performance consideration, but I think it's worth mentioning. -- https://bugs.ruby-lang.org/
Issue #20594 has been updated by shugo (Shugo Maeda). Eregon (Benoit Daloze) wrote in #note-11:
I wonder if that's a bug of `bytesplice`. It's not like e.g. `setbyte` would respect the encoding (it cannot possibly). OTOH `byteindex` also takes a String argument and raises `Encoding::CompatibilityError` for `"é".b.byteindex "é"` so I guess it's not so clear-cut currently (I get the replies above a bit better).
It's intended behavior. Having a long history of suffering from character encoding issues such as mojibake, we Japanese believe that operations on strings with different encodings should be approached with caution. `byte-` methods could be considered exceptions, but at present, they are not. That's why Matz doesn't like the name `byteconcat`. ---------------------------------------- Feature #20594: A new String method to append bytes while preserving encoding https://bugs.ruby-lang.org/issues/20594#change-109101 * Author: byroot (Jean Boussier) * Status: Open * Assignee: byroot (Jean Boussier) ---------------------------------------- ### Context When working with binary protocols such as `protobuf` or `MessagePack`, you may often need to assemble multiple strings of different encoding: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title << 255 << body.bytesize << body end end Post.new("Hello", "World").serialize("somedata".b) # => "somedata\xFF\x05Hello\xFF\x05World" #<Encoding:ASCII-8BIT> ``` The problem in the above case, is that because `Encoding::ASCII_8BIT` is declared as ASCII compatible, if one of the appended string contains bytes outside the ASCII range, string is automatically promoted to another encoding, which then leads to encoding issues: ```ruby Post.new("H€llo", "Wôrld").serialize("somedata".b) # => incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError) ``` In many cases, you want to append to a String without changing the receiver's encoding. The issue isn't exclusive to binary protocols and formats, it also happen with ASCII protocols that accept arbitrary bytes inline, like Redis's RESP protocol or even HTTP/1.1. ### Previous discussion There was a similar feature request a while ago, but it was abandoned: https://bugs.ruby-lang.org/issues/14975 ### Existing solutions You can of course always cast the strings you append to avoid this problem: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title.b << 255 << body.bytesize << body.b end end ``` But this cause a lot of needless allocations. You'd think you could also use `bytesplice`, but it actually has the same issue: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize buf.bytesplice(buf.bytesize, title.bytesize, title) buf << 255 << body.bytesize buf.bytesplice(buf.bytesize, body.bytesize, title) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => 'String#bytesplice': incompatible character encodings: BINARY (ASCII-8BIT) and UTF-8 (Encoding::CompatibilityError) ``` And even if it worked, it would be very unergonomic. ### Proposal: a `byteconcat` method A solution to this would be to add a new `byteconcat` method, that could be shimed as: ```ruby class String def byteconcat(*strings) strings.map! do |s| if s.is_a?(String) && s.encoding != encoding s.dup.force_encoding(encoding) else s end end concat(*strings) end end Post = Struct.new(:title, :body) do def serialize(buf) buf.byteconcat( 255, title.bytesize, title, 255, body.bytesize, body, ) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => "somedata\xFF\aH\xE2\x82\xACllo\xFF\x06W\xC3\xB4rld" #<Encoding:ASCII-8BIT> ``` But of course a builtin implementation wouldn't need to dup the arguments. Like other `byte*` methods, it's the responsibility of the caller to ensure the resulting string has a valid encoding, or to deal with it if not. ### Method name and signature #### Name This proposal suggests `String#byteconcat`, to mirror `String#concat`, but other names are possible: - `byteappend` (like `Array#append`) - `bytepush` (like `Array#push`) #### Signature This proposal makes `byteconcat` accept either `String` or `Integer` (in char range) arguments like `concat`. I believe it makes sense for consistency and also because it's not uncommon for protocols to have some byte based segments, and Integers are more convenient there. The proposed method also accept variable arguments for consistency with `String#concat`, `Array#push`, `Array#append`. The proposed method returns self, like `concat` and others. ### YJIT consideration I consulted @maximecb about this proposal, and according to her, accepting variable arguments makes it harder for YJIT to optimize. I suspect consistency with other APIs trumps the performance consideration, but I think it's worth mentioning. -- https://bugs.ruby-lang.org/
Issue #20594 has been updated by Dan0042 (Daniel DeLorme). From the [dev meeting log](https://github.com/ruby/dev-meeting-log/blob/master/2024/DevMeeting-2024-07-...) these two points stuck out at me: * nobu: can we use IO::Buffer? * shyouhei: why not StringIO? These are good questions that I also wondered about, but the answers are not recorded in the meeting log. What is the reason not to use these APIs which are pretty much designed for this exact use case? Why introduce a third API? BTW I have to say I find it extremely ironic that we went to so much trouble to migrate from byte-oriented strings in 1.8 to character-oriented strings in 1.9, and now we're re-adding byte-oriented methods. ---------------------------------------- Feature #20594: A new String method to append bytes while preserving encoding https://bugs.ruby-lang.org/issues/20594#change-109159 * Author: byroot (Jean Boussier) * Status: Open * Assignee: byroot (Jean Boussier) ---------------------------------------- ### Context When working with binary protocols such as `protobuf` or `MessagePack`, you may often need to assemble multiple strings of different encoding: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title << 255 << body.bytesize << body end end Post.new("Hello", "World").serialize("somedata".b) # => "somedata\xFF\x05Hello\xFF\x05World" #<Encoding:ASCII-8BIT> ``` The problem in the above case, is that because `Encoding::ASCII_8BIT` is declared as ASCII compatible, if one of the appended string contains bytes outside the ASCII range, string is automatically promoted to another encoding, which then leads to encoding issues: ```ruby Post.new("H€llo", "Wôrld").serialize("somedata".b) # => incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError) ``` In many cases, you want to append to a String without changing the receiver's encoding. The issue isn't exclusive to binary protocols and formats, it also happen with ASCII protocols that accept arbitrary bytes inline, like Redis's RESP protocol or even HTTP/1.1. ### Previous discussion There was a similar feature request a while ago, but it was abandoned: https://bugs.ruby-lang.org/issues/14975 ### Existing solutions You can of course always cast the strings you append to avoid this problem: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title.b << 255 << body.bytesize << body.b end end ``` But this cause a lot of needless allocations. You'd think you could also use `bytesplice`, but it actually has the same issue: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize buf.bytesplice(buf.bytesize, title.bytesize, title) buf << 255 << body.bytesize buf.bytesplice(buf.bytesize, body.bytesize, title) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => 'String#bytesplice': incompatible character encodings: BINARY (ASCII-8BIT) and UTF-8 (Encoding::CompatibilityError) ``` And even if it worked, it would be very unergonomic. ### Proposal: a `byteconcat` method A solution to this would be to add a new `byteconcat` method, that could be shimed as: ```ruby class String def byteconcat(*strings) strings.map! do |s| if s.is_a?(String) && s.encoding != encoding s.dup.force_encoding(encoding) else s end end concat(*strings) end end Post = Struct.new(:title, :body) do def serialize(buf) buf.byteconcat( 255, title.bytesize, title, 255, body.bytesize, body, ) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => "somedata\xFF\aH\xE2\x82\xACllo\xFF\x06W\xC3\xB4rld" #<Encoding:ASCII-8BIT> ``` But of course a builtin implementation wouldn't need to dup the arguments. Like other `byte*` methods, it's the responsibility of the caller to ensure the resulting string has a valid encoding, or to deal with it if not. ### Method name and signature #### Name This proposal suggests `String#byteconcat`, to mirror `String#concat`, but other names are possible: - `byteappend` (like `Array#append`) - `bytepush` (like `Array#push`) #### Signature This proposal makes `byteconcat` accept either `String` or `Integer` (in char range) arguments like `concat`. I believe it makes sense for consistency and also because it's not uncommon for protocols to have some byte based segments, and Integers are more convenient there. The proposed method also accept variable arguments for consistency with `String#concat`, `Array#push`, `Array#append`. The proposed method returns self, like `concat` and others. ### YJIT consideration I consulted @maximecb about this proposal, and according to her, accepting variable arguments makes it harder for YJIT to optimize. I suspect consistency with other APIs trumps the performance consideration, but I think it's worth mentioning. -- https://bugs.ruby-lang.org/
Issue #20594 has been updated by alanwu (Alan Wu).
What is the reason not to use these APIs which are pretty much designed for this exact use case? Why introduce a third API?
Sometimes only `String` gives you the most convenience and efficiency because of API interoperability. For example, binary protocols often have you reaching for Array#pack and String#unpack, and these don't directly work with StringIO and IO::Buffer. ---------------------------------------- Feature #20594: A new String method to append bytes while preserving encoding https://bugs.ruby-lang.org/issues/20594#change-109160 * Author: byroot (Jean Boussier) * Status: Open * Assignee: byroot (Jean Boussier) ---------------------------------------- ### Context When working with binary protocols such as `protobuf` or `MessagePack`, you may often need to assemble multiple strings of different encoding: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title << 255 << body.bytesize << body end end Post.new("Hello", "World").serialize("somedata".b) # => "somedata\xFF\x05Hello\xFF\x05World" #<Encoding:ASCII-8BIT> ``` The problem in the above case, is that because `Encoding::ASCII_8BIT` is declared as ASCII compatible, if one of the appended string contains bytes outside the ASCII range, string is automatically promoted to another encoding, which then leads to encoding issues: ```ruby Post.new("H€llo", "Wôrld").serialize("somedata".b) # => incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError) ``` In many cases, you want to append to a String without changing the receiver's encoding. The issue isn't exclusive to binary protocols and formats, it also happen with ASCII protocols that accept arbitrary bytes inline, like Redis's RESP protocol or even HTTP/1.1. ### Previous discussion There was a similar feature request a while ago, but it was abandoned: https://bugs.ruby-lang.org/issues/14975 ### Existing solutions You can of course always cast the strings you append to avoid this problem: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title.b << 255 << body.bytesize << body.b end end ``` But this cause a lot of needless allocations. You'd think you could also use `bytesplice`, but it actually has the same issue: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize buf.bytesplice(buf.bytesize, title.bytesize, title) buf << 255 << body.bytesize buf.bytesplice(buf.bytesize, body.bytesize, title) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => 'String#bytesplice': incompatible character encodings: BINARY (ASCII-8BIT) and UTF-8 (Encoding::CompatibilityError) ``` And even if it worked, it would be very unergonomic. ### Proposal: a `byteconcat` method A solution to this would be to add a new `byteconcat` method, that could be shimed as: ```ruby class String def byteconcat(*strings) strings.map! do |s| if s.is_a?(String) && s.encoding != encoding s.dup.force_encoding(encoding) else s end end concat(*strings) end end Post = Struct.new(:title, :body) do def serialize(buf) buf.byteconcat( 255, title.bytesize, title, 255, body.bytesize, body, ) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => "somedata\xFF\aH\xE2\x82\xACllo\xFF\x06W\xC3\xB4rld" #<Encoding:ASCII-8BIT> ``` But of course a builtin implementation wouldn't need to dup the arguments. Like other `byte*` methods, it's the responsibility of the caller to ensure the resulting string has a valid encoding, or to deal with it if not. ### Method name and signature #### Name This proposal suggests `String#byteconcat`, to mirror `String#concat`, but other names are possible: - `byteappend` (like `Array#append`) - `bytepush` (like `Array#push`) #### Signature This proposal makes `byteconcat` accept either `String` or `Integer` (in char range) arguments like `concat`. I believe it makes sense for consistency and also because it's not uncommon for protocols to have some byte based segments, and Integers are more convenient there. The proposed method also accept variable arguments for consistency with `String#concat`, `Array#push`, `Array#append`. The proposed method returns self, like `concat` and others. ### YJIT consideration I consulted @maximecb about this proposal, and according to her, accepting variable arguments makes it harder for YJIT to optimize. I suspect consistency with other APIs trumps the performance consideration, but I think it's worth mentioning. -- https://bugs.ruby-lang.org/
Issue #20594 has been updated by byroot (Jean Boussier). @Dan0042 I already answered your same question in [Feature #20394]. Whether we like it or not neither StringIO nor IO::Buffer are fast, compatible and convenient enough to replace `String` for these use case. If you don't believe me, I encourage you to try using either in https://github.com/Shopify/protoboeuf or https://github.com/redis-rb/redis-client (or any other gem of your choice) and see how that goes. ---------------------------------------- Feature #20594: A new String method to append bytes while preserving encoding https://bugs.ruby-lang.org/issues/20594#change-109163 * Author: byroot (Jean Boussier) * Status: Open * Assignee: byroot (Jean Boussier) ---------------------------------------- ### Context When working with binary protocols such as `protobuf` or `MessagePack`, you may often need to assemble multiple strings of different encoding: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title << 255 << body.bytesize << body end end Post.new("Hello", "World").serialize("somedata".b) # => "somedata\xFF\x05Hello\xFF\x05World" #<Encoding:ASCII-8BIT> ``` The problem in the above case, is that because `Encoding::ASCII_8BIT` is declared as ASCII compatible, if one of the appended string contains bytes outside the ASCII range, string is automatically promoted to another encoding, which then leads to encoding issues: ```ruby Post.new("H€llo", "Wôrld").serialize("somedata".b) # => incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError) ``` In many cases, you want to append to a String without changing the receiver's encoding. The issue isn't exclusive to binary protocols and formats, it also happen with ASCII protocols that accept arbitrary bytes inline, like Redis's RESP protocol or even HTTP/1.1. ### Previous discussion There was a similar feature request a while ago, but it was abandoned: https://bugs.ruby-lang.org/issues/14975 ### Existing solutions You can of course always cast the strings you append to avoid this problem: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title.b << 255 << body.bytesize << body.b end end ``` But this cause a lot of needless allocations. You'd think you could also use `bytesplice`, but it actually has the same issue: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize buf.bytesplice(buf.bytesize, title.bytesize, title) buf << 255 << body.bytesize buf.bytesplice(buf.bytesize, body.bytesize, title) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => 'String#bytesplice': incompatible character encodings: BINARY (ASCII-8BIT) and UTF-8 (Encoding::CompatibilityError) ``` And even if it worked, it would be very unergonomic. ### Proposal: a `byteconcat` method A solution to this would be to add a new `byteconcat` method, that could be shimed as: ```ruby class String def byteconcat(*strings) strings.map! do |s| if s.is_a?(String) && s.encoding != encoding s.dup.force_encoding(encoding) else s end end concat(*strings) end end Post = Struct.new(:title, :body) do def serialize(buf) buf.byteconcat( 255, title.bytesize, title, 255, body.bytesize, body, ) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => "somedata\xFF\aH\xE2\x82\xACllo\xFF\x06W\xC3\xB4rld" #<Encoding:ASCII-8BIT> ``` But of course a builtin implementation wouldn't need to dup the arguments. Like other `byte*` methods, it's the responsibility of the caller to ensure the resulting string has a valid encoding, or to deal with it if not. ### Method name and signature #### Name This proposal suggests `String#byteconcat`, to mirror `String#concat`, but other names are possible: - `byteappend` (like `Array#append`) - `bytepush` (like `Array#push`) #### Signature This proposal makes `byteconcat` accept either `String` or `Integer` (in char range) arguments like `concat`. I believe it makes sense for consistency and also because it's not uncommon for protocols to have some byte based segments, and Integers are more convenient there. The proposed method also accept variable arguments for consistency with `String#concat`, `Array#push`, `Array#append`. The proposed method returns self, like `concat` and others. ### YJIT consideration I consulted @maximecb about this proposal, and according to her, accepting variable arguments makes it harder for YJIT to optimize. I suspect consistency with other APIs trumps the performance consideration, but I think it's worth mentioning. -- https://bugs.ruby-lang.org/
Issue #20594 has been updated by byroot (Jean Boussier). Ok, so after thinking about this for a bit, I think a good name would be: - `String#append_bytes(String) => self` - `String#append_byte(Integer) => self` It's still mentioning bytes, but it's not using the same `byte*` prefix as other methods, so I think it's different enough. If anything mentioning byte is deemed to confusing I can search or something else, but I really think that's the proper name for the concept. ---------------------------------------- Feature #20594: A new String method to append bytes while preserving encoding https://bugs.ruby-lang.org/issues/20594#change-109226 * Author: byroot (Jean Boussier) * Status: Open * Assignee: byroot (Jean Boussier) ---------------------------------------- ### Context When working with binary protocols such as `protobuf` or `MessagePack`, you may often need to assemble multiple strings of different encoding: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title << 255 << body.bytesize << body end end Post.new("Hello", "World").serialize("somedata".b) # => "somedata\xFF\x05Hello\xFF\x05World" #<Encoding:ASCII-8BIT> ``` The problem in the above case, is that because `Encoding::ASCII_8BIT` is declared as ASCII compatible, if one of the appended string contains bytes outside the ASCII range, string is automatically promoted to another encoding, which then leads to encoding issues: ```ruby Post.new("H€llo", "Wôrld").serialize("somedata".b) # => incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError) ``` In many cases, you want to append to a String without changing the receiver's encoding. The issue isn't exclusive to binary protocols and formats, it also happen with ASCII protocols that accept arbitrary bytes inline, like Redis's RESP protocol or even HTTP/1.1. ### Previous discussion There was a similar feature request a while ago, but it was abandoned: https://bugs.ruby-lang.org/issues/14975 ### Existing solutions You can of course always cast the strings you append to avoid this problem: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title.b << 255 << body.bytesize << body.b end end ``` But this cause a lot of needless allocations. You'd think you could also use `bytesplice`, but it actually has the same issue: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize buf.bytesplice(buf.bytesize, title.bytesize, title) buf << 255 << body.bytesize buf.bytesplice(buf.bytesize, body.bytesize, title) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => 'String#bytesplice': incompatible character encodings: BINARY (ASCII-8BIT) and UTF-8 (Encoding::CompatibilityError) ``` And even if it worked, it would be very unergonomic. ### Proposal: a `byteconcat` method A solution to this would be to add a new `byteconcat` method, that could be shimed as: ```ruby class String def byteconcat(*strings) strings.map! do |s| if s.is_a?(String) && s.encoding != encoding s.dup.force_encoding(encoding) else s end end concat(*strings) end end Post = Struct.new(:title, :body) do def serialize(buf) buf.byteconcat( 255, title.bytesize, title, 255, body.bytesize, body, ) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => "somedata\xFF\aH\xE2\x82\xACllo\xFF\x06W\xC3\xB4rld" #<Encoding:ASCII-8BIT> ``` But of course a builtin implementation wouldn't need to dup the arguments. Like other `byte*` methods, it's the responsibility of the caller to ensure the resulting string has a valid encoding, or to deal with it if not. ### Method name and signature #### Name This proposal suggests `String#byteconcat`, to mirror `String#concat`, but other names are possible: - `byteappend` (like `Array#append`) - `bytepush` (like `Array#push`) #### Signature This proposal makes `byteconcat` accept either `String` or `Integer` (in char range) arguments like `concat`. I believe it makes sense for consistency and also because it's not uncommon for protocols to have some byte based segments, and Integers are more convenient there. The proposed method also accept variable arguments for consistency with `String#concat`, `Array#push`, `Array#append`. The proposed method returns self, like `concat` and others. ### YJIT consideration I consulted @maximecb about this proposal, and according to her, accepting variable arguments makes it harder for YJIT to optimize. I suspect consistency with other APIs trumps the performance consideration, but I think it's worth mentioning. -- https://bugs.ruby-lang.org/
Issue #20594 has been updated by duerst (Martin Dürst). Eregon (Benoit Daloze) wrote in #note-9:
My understanding of `byte*` methods is they treat the String as a byte array, which implies indices are just byte indices but also that the encoding is ignored (it seems clear when one does `"é".getbyte(0)`).
This may need a completely separate issue, but when I introduced `String#force_encoding`, I was imagining adding a block to it so that the forced encoding would only apply inside the block. ```Ruby s = "your string (e.g. UTF-8 or whatever) here" s.force_encoding(Encoding::BINARY) { # binary operations go here } s.encoding # => Encoding:UTF-8 ``` This would streamline things quite a bit; no need for separate `byte*` methods. Using a block should also help because in many use cases, there's be more than just one `byte*` method. The question then would of course be how to optimize things. But to a large part, the optimizations are actually already done, because different encodings use different primitives. ---------------------------------------- Feature #20594: A new String method to append bytes while preserving encoding https://bugs.ruby-lang.org/issues/20594#change-109274 * Author: byroot (Jean Boussier) * Status: Open * Assignee: byroot (Jean Boussier) ---------------------------------------- ### Context When working with binary protocols such as `protobuf` or `MessagePack`, you may often need to assemble multiple strings of different encoding: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title << 255 << body.bytesize << body end end Post.new("Hello", "World").serialize("somedata".b) # => "somedata\xFF\x05Hello\xFF\x05World" #<Encoding:ASCII-8BIT> ``` The problem in the above case, is that because `Encoding::ASCII_8BIT` is declared as ASCII compatible, if one of the appended string contains bytes outside the ASCII range, string is automatically promoted to another encoding, which then leads to encoding issues: ```ruby Post.new("H€llo", "Wôrld").serialize("somedata".b) # => incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError) ``` In many cases, you want to append to a String without changing the receiver's encoding. The issue isn't exclusive to binary protocols and formats, it also happen with ASCII protocols that accept arbitrary bytes inline, like Redis's RESP protocol or even HTTP/1.1. ### Previous discussion There was a similar feature request a while ago, but it was abandoned: https://bugs.ruby-lang.org/issues/14975 ### Existing solutions You can of course always cast the strings you append to avoid this problem: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title.b << 255 << body.bytesize << body.b end end ``` But this cause a lot of needless allocations. You'd think you could also use `bytesplice`, but it actually has the same issue: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize buf.bytesplice(buf.bytesize, title.bytesize, title) buf << 255 << body.bytesize buf.bytesplice(buf.bytesize, body.bytesize, title) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => 'String#bytesplice': incompatible character encodings: BINARY (ASCII-8BIT) and UTF-8 (Encoding::CompatibilityError) ``` And even if it worked, it would be very unergonomic. ### Proposal: a `byteconcat` method A solution to this would be to add a new `byteconcat` method, that could be shimed as: ```ruby class String def byteconcat(*strings) strings.map! do |s| if s.is_a?(String) && s.encoding != encoding s.dup.force_encoding(encoding) else s end end concat(*strings) end end Post = Struct.new(:title, :body) do def serialize(buf) buf.byteconcat( 255, title.bytesize, title, 255, body.bytesize, body, ) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => "somedata\xFF\aH\xE2\x82\xACllo\xFF\x06W\xC3\xB4rld" #<Encoding:ASCII-8BIT> ``` But of course a builtin implementation wouldn't need to dup the arguments. Like other `byte*` methods, it's the responsibility of the caller to ensure the resulting string has a valid encoding, or to deal with it if not. ### Method name and signature #### Name This proposal suggests `String#byteconcat`, to mirror `String#concat`, but other names are possible: - `byteappend` (like `Array#append`) - `bytepush` (like `Array#push`) #### Signature This proposal makes `byteconcat` accept either `String` or `Integer` (in char range) arguments like `concat`. I believe it makes sense for consistency and also because it's not uncommon for protocols to have some byte based segments, and Integers are more convenient there. The proposed method also accept variable arguments for consistency with `String#concat`, `Array#push`, `Array#append`. The proposed method returns self, like `concat` and others. ### YJIT consideration I consulted @maximecb about this proposal, and according to her, accepting variable arguments makes it harder for YJIT to optimize. I suspect consistency with other APIs trumps the performance consideration, but I think it's worth mentioning. -- https://bugs.ruby-lang.org/
Issue #20594 has been updated by Dan0042 (Daniel DeLorme). duerst (Martin Dürst) wrote in #note-17:
This may need a completely separate issue, but when I introduced `String#force_encoding`, I was imagining adding a block to it so that the forced encoding would only apply inside the block.
+1, that would be very convenient, but yes it's a different issue because it doesn't ignore Encoding::CompatibilityError which is the point of this proposal. Many languages have a ByteArray or ByteBuffer separate from String, and I believe this is what ruby needs. Design-wise, in the long run, I believe it's better to evolve StringIO or IO::Buffer into a proper bytebuffer to handle encoding-agnostic bytes; and String should only handle encoding-aware characters. (I apologize for beating a dead horse.) ---------------------------------------- Feature #20594: A new String method to append bytes while preserving encoding https://bugs.ruby-lang.org/issues/20594#change-109276 * Author: byroot (Jean Boussier) * Status: Open * Assignee: byroot (Jean Boussier) ---------------------------------------- ### Context When working with binary protocols such as `protobuf` or `MessagePack`, you may often need to assemble multiple strings of different encoding: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title << 255 << body.bytesize << body end end Post.new("Hello", "World").serialize("somedata".b) # => "somedata\xFF\x05Hello\xFF\x05World" #<Encoding:ASCII-8BIT> ``` The problem in the above case, is that because `Encoding::ASCII_8BIT` is declared as ASCII compatible, if one of the appended string contains bytes outside the ASCII range, string is automatically promoted to another encoding, which then leads to encoding issues: ```ruby Post.new("H€llo", "Wôrld").serialize("somedata".b) # => incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError) ``` In many cases, you want to append to a String without changing the receiver's encoding. The issue isn't exclusive to binary protocols and formats, it also happen with ASCII protocols that accept arbitrary bytes inline, like Redis's RESP protocol or even HTTP/1.1. ### Previous discussion There was a similar feature request a while ago, but it was abandoned: https://bugs.ruby-lang.org/issues/14975 ### Existing solutions You can of course always cast the strings you append to avoid this problem: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title.b << 255 << body.bytesize << body.b end end ``` But this cause a lot of needless allocations. You'd think you could also use `bytesplice`, but it actually has the same issue: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize buf.bytesplice(buf.bytesize, title.bytesize, title) buf << 255 << body.bytesize buf.bytesplice(buf.bytesize, body.bytesize, title) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => 'String#bytesplice': incompatible character encodings: BINARY (ASCII-8BIT) and UTF-8 (Encoding::CompatibilityError) ``` And even if it worked, it would be very unergonomic. ### Proposal: a `byteconcat` method A solution to this would be to add a new `byteconcat` method, that could be shimed as: ```ruby class String def byteconcat(*strings) strings.map! do |s| if s.is_a?(String) && s.encoding != encoding s.dup.force_encoding(encoding) else s end end concat(*strings) end end Post = Struct.new(:title, :body) do def serialize(buf) buf.byteconcat( 255, title.bytesize, title, 255, body.bytesize, body, ) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => "somedata\xFF\aH\xE2\x82\xACllo\xFF\x06W\xC3\xB4rld" #<Encoding:ASCII-8BIT> ``` But of course a builtin implementation wouldn't need to dup the arguments. Like other `byte*` methods, it's the responsibility of the caller to ensure the resulting string has a valid encoding, or to deal with it if not. ### Method name and signature #### Name This proposal suggests `String#byteconcat`, to mirror `String#concat`, but other names are possible: - `byteappend` (like `Array#append`) - `bytepush` (like `Array#push`) #### Signature This proposal makes `byteconcat` accept either `String` or `Integer` (in char range) arguments like `concat`. I believe it makes sense for consistency and also because it's not uncommon for protocols to have some byte based segments, and Integers are more convenient there. The proposed method also accept variable arguments for consistency with `String#concat`, `Array#push`, `Array#append`. The proposed method returns self, like `concat` and others. ### YJIT consideration I consulted @maximecb about this proposal, and according to her, accepting variable arguments makes it harder for YJIT to optimize. I suspect consistency with other APIs trumps the performance consideration, but I think it's worth mentioning. -- https://bugs.ruby-lang.org/
Issue #20594 has been updated by matz (Yukihiro Matsumoto). `append_bytes` seems OK for me. Could you design the concrete behavior of the method: * does it take more than one argument? * does it take integers too? * what is the result of the method encoding-wise? * etc. Matz. ---------------------------------------- Feature #20594: A new String method to append bytes while preserving encoding https://bugs.ruby-lang.org/issues/20594#change-109315 * Author: byroot (Jean Boussier) * Status: Open * Assignee: byroot (Jean Boussier) ---------------------------------------- ### Context When working with binary protocols such as `protobuf` or `MessagePack`, you may often need to assemble multiple strings of different encoding: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title << 255 << body.bytesize << body end end Post.new("Hello", "World").serialize("somedata".b) # => "somedata\xFF\x05Hello\xFF\x05World" #<Encoding:ASCII-8BIT> ``` The problem in the above case, is that because `Encoding::ASCII_8BIT` is declared as ASCII compatible, if one of the appended string contains bytes outside the ASCII range, string is automatically promoted to another encoding, which then leads to encoding issues: ```ruby Post.new("H€llo", "Wôrld").serialize("somedata".b) # => incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError) ``` In many cases, you want to append to a String without changing the receiver's encoding. The issue isn't exclusive to binary protocols and formats, it also happen with ASCII protocols that accept arbitrary bytes inline, like Redis's RESP protocol or even HTTP/1.1. ### Previous discussion There was a similar feature request a while ago, but it was abandoned: https://bugs.ruby-lang.org/issues/14975 ### Existing solutions You can of course always cast the strings you append to avoid this problem: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title.b << 255 << body.bytesize << body.b end end ``` But this cause a lot of needless allocations. You'd think you could also use `bytesplice`, but it actually has the same issue: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize buf.bytesplice(buf.bytesize, title.bytesize, title) buf << 255 << body.bytesize buf.bytesplice(buf.bytesize, body.bytesize, title) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => 'String#bytesplice': incompatible character encodings: BINARY (ASCII-8BIT) and UTF-8 (Encoding::CompatibilityError) ``` And even if it worked, it would be very unergonomic. ### Proposal: a `byteconcat` method A solution to this would be to add a new `byteconcat` method, that could be shimed as: ```ruby class String def byteconcat(*strings) strings.map! do |s| if s.is_a?(String) && s.encoding != encoding s.dup.force_encoding(encoding) else s end end concat(*strings) end end Post = Struct.new(:title, :body) do def serialize(buf) buf.byteconcat( 255, title.bytesize, title, 255, body.bytesize, body, ) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => "somedata\xFF\aH\xE2\x82\xACllo\xFF\x06W\xC3\xB4rld" #<Encoding:ASCII-8BIT> ``` But of course a builtin implementation wouldn't need to dup the arguments. Like other `byte*` methods, it's the responsibility of the caller to ensure the resulting string has a valid encoding, or to deal with it if not. ### Method name and signature #### Name This proposal suggests `String#byteconcat`, to mirror `String#concat`, but other names are possible: - `byteappend` (like `Array#append`) - `bytepush` (like `Array#push`) #### Signature This proposal makes `byteconcat` accept either `String` or `Integer` (in char range) arguments like `concat`. I believe it makes sense for consistency and also because it's not uncommon for protocols to have some byte based segments, and Integers are more convenient there. The proposed method also accept variable arguments for consistency with `String#concat`, `Array#push`, `Array#append`. The proposed method returns self, like `concat` and others. ### YJIT consideration I consulted @maximecb about this proposal, and according to her, accepting variable arguments makes it harder for YJIT to optimize. I suspect consistency with other APIs trumps the performance consideration, but I think it's worth mentioning. -- https://bugs.ruby-lang.org/
Issue #20594 has been updated by byroot (Jean Boussier). Thank you Matz. I opened a PR that implement the method as envisioned: https://github.com/ruby/ruby/pull/11293
does it take more than one argument?
No, a single argument, and only a String (T_STRING) as suggested by YJIT people to make it optimizable.
does it take integers too?
Similarly, YJIT people suggested to not take different types. Hence why I proposed two methods `append_byte(Integer)` and `append_bytes(String)`.
what is the result of the method encoding-wise?
The receiver encoding is never changed, even if it means that it's encoding become invalid. It's the called responsability to check `String#valid_encoding?` if that's a possibility and to deal with it. It also means `append_bytes` never raises an `Encoding::CompatibilityError`. ---------------------------------------- Feature #20594: A new String method to append bytes while preserving encoding https://bugs.ruby-lang.org/issues/20594#change-109318 * Author: byroot (Jean Boussier) * Status: Open * Assignee: byroot (Jean Boussier) ---------------------------------------- ### Context When working with binary protocols such as `protobuf` or `MessagePack`, you may often need to assemble multiple strings of different encoding: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title << 255 << body.bytesize << body end end Post.new("Hello", "World").serialize("somedata".b) # => "somedata\xFF\x05Hello\xFF\x05World" #<Encoding:ASCII-8BIT> ``` The problem in the above case, is that because `Encoding::ASCII_8BIT` is declared as ASCII compatible, if one of the appended string contains bytes outside the ASCII range, string is automatically promoted to another encoding, which then leads to encoding issues: ```ruby Post.new("H€llo", "Wôrld").serialize("somedata".b) # => incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError) ``` In many cases, you want to append to a String without changing the receiver's encoding. The issue isn't exclusive to binary protocols and formats, it also happen with ASCII protocols that accept arbitrary bytes inline, like Redis's RESP protocol or even HTTP/1.1. ### Previous discussion There was a similar feature request a while ago, but it was abandoned: https://bugs.ruby-lang.org/issues/14975 ### Existing solutions You can of course always cast the strings you append to avoid this problem: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title.b << 255 << body.bytesize << body.b end end ``` But this cause a lot of needless allocations. You'd think you could also use `bytesplice`, but it actually has the same issue: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize buf.bytesplice(buf.bytesize, title.bytesize, title) buf << 255 << body.bytesize buf.bytesplice(buf.bytesize, body.bytesize, title) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => 'String#bytesplice': incompatible character encodings: BINARY (ASCII-8BIT) and UTF-8 (Encoding::CompatibilityError) ``` And even if it worked, it would be very unergonomic. ### Proposal: a `byteconcat` method A solution to this would be to add a new `byteconcat` method, that could be shimed as: ```ruby class String def byteconcat(*strings) strings.map! do |s| if s.is_a?(String) && s.encoding != encoding s.dup.force_encoding(encoding) else s end end concat(*strings) end end Post = Struct.new(:title, :body) do def serialize(buf) buf.byteconcat( 255, title.bytesize, title, 255, body.bytesize, body, ) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => "somedata\xFF\aH\xE2\x82\xACllo\xFF\x06W\xC3\xB4rld" #<Encoding:ASCII-8BIT> ``` But of course a builtin implementation wouldn't need to dup the arguments. Like other `byte*` methods, it's the responsibility of the caller to ensure the resulting string has a valid encoding, or to deal with it if not. ### Method name and signature #### Name This proposal suggests `String#byteconcat`, to mirror `String#concat`, but other names are possible: - `byteappend` (like `Array#append`) - `bytepush` (like `Array#push`) #### Signature This proposal makes `byteconcat` accept either `String` or `Integer` (in char range) arguments like `concat`. I believe it makes sense for consistency and also because it's not uncommon for protocols to have some byte based segments, and Integers are more convenient there. The proposed method also accept variable arguments for consistency with `String#concat`, `Array#push`, `Array#append`. The proposed method returns self, like `concat` and others. ### YJIT consideration I consulted @maximecb about this proposal, and according to her, accepting variable arguments makes it harder for YJIT to optimize. I suspect consistency with other APIs trumps the performance consideration, but I think it's worth mentioning. -- https://bugs.ruby-lang.org/
Issue #20594 has been updated by alanwu (Alan Wu). Let me present an alternative design that only adds one method. The name is String#append_as_bytes, and the name provides a framing of "reinterpretation" that helps to explain the behavior of the method. ``` call-seq: append_as_bytes(*objects) -> self Interpret arguments as bytes and append them to +self+ without changing the encoding of +self+. For each object that is a String, append the bytes of the string to +self+. For each Integer object +i+, append a byte that is the bitwise AND of +i+ and +0xff+. If any other type of objects is in +objects+, leave +self+ unmodified and raise ArgumentError. This method does not attempt to implicitly convert any arguments. Examples: 7z_signature = ''.b 7z_signature.append_bytes('7z', 0xbc, 0xaf, 0x27, 0x1c) #=> "7z\xBC\xAF'\x1C" ``` It's clear from the name that the method has its own interpretation of arguments. That gives a hint that it does something unusual, as it breaks away from the default, "bytes with an encoding" stance of strings. It also grammatically stands out from existing `*byte*` methods, a reflection of the differences in behavior. For string arguments, it's the same as `append_bytes(String)` from byroot. For integers, the `i & 0xff` masking behavior comes from String#setbyte. Note that it masks without making calls. ```ruby -128 & 0xff # => 128 "x".tap{ _1.setbyte(0, -128)}.bytes # => [128] ``` This masking is how it interprets an integer as a byte. The method does not accept arrays for simplicity, as splatting is already available as a flexible option for callers. I think this design strikes a good balance between usability, efficiency, and how well compilers can handle it. ---------------------------------------- Feature #20594: A new String method to append bytes while preserving encoding https://bugs.ruby-lang.org/issues/20594#change-109332 * Author: byroot (Jean Boussier) * Status: Open * Assignee: byroot (Jean Boussier) ---------------------------------------- ### Context When working with binary protocols such as `protobuf` or `MessagePack`, you may often need to assemble multiple strings of different encoding: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title << 255 << body.bytesize << body end end Post.new("Hello", "World").serialize("somedata".b) # => "somedata\xFF\x05Hello\xFF\x05World" #<Encoding:ASCII-8BIT> ``` The problem in the above case, is that because `Encoding::ASCII_8BIT` is declared as ASCII compatible, if one of the appended string contains bytes outside the ASCII range, string is automatically promoted to another encoding, which then leads to encoding issues: ```ruby Post.new("H€llo", "Wôrld").serialize("somedata".b) # => incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError) ``` In many cases, you want to append to a String without changing the receiver's encoding. The issue isn't exclusive to binary protocols and formats, it also happen with ASCII protocols that accept arbitrary bytes inline, like Redis's RESP protocol or even HTTP/1.1. ### Previous discussion There was a similar feature request a while ago, but it was abandoned: https://bugs.ruby-lang.org/issues/14975 ### Existing solutions You can of course always cast the strings you append to avoid this problem: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title.b << 255 << body.bytesize << body.b end end ``` But this cause a lot of needless allocations. You'd think you could also use `bytesplice`, but it actually has the same issue: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize buf.bytesplice(buf.bytesize, title.bytesize, title) buf << 255 << body.bytesize buf.bytesplice(buf.bytesize, body.bytesize, title) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => 'String#bytesplice': incompatible character encodings: BINARY (ASCII-8BIT) and UTF-8 (Encoding::CompatibilityError) ``` And even if it worked, it would be very unergonomic. ### Proposal: a `byteconcat` method A solution to this would be to add a new `byteconcat` method, that could be shimed as: ```ruby class String def byteconcat(*strings) strings.map! do |s| if s.is_a?(String) && s.encoding != encoding s.dup.force_encoding(encoding) else s end end concat(*strings) end end Post = Struct.new(:title, :body) do def serialize(buf) buf.byteconcat( 255, title.bytesize, title, 255, body.bytesize, body, ) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => "somedata\xFF\aH\xE2\x82\xACllo\xFF\x06W\xC3\xB4rld" #<Encoding:ASCII-8BIT> ``` But of course a builtin implementation wouldn't need to dup the arguments. Like other `byte*` methods, it's the responsibility of the caller to ensure the resulting string has a valid encoding, or to deal with it if not. ### Method name and signature #### Name This proposal suggests `String#byteconcat`, to mirror `String#concat`, but other names are possible: - `byteappend` (like `Array#append`) - `bytepush` (like `Array#push`) #### Signature This proposal makes `byteconcat` accept either `String` or `Integer` (in char range) arguments like `concat`. I believe it makes sense for consistency and also because it's not uncommon for protocols to have some byte based segments, and Integers are more convenient there. The proposed method also accept variable arguments for consistency with `String#concat`, `Array#push`, `Array#append`. The proposed method returns self, like `concat` and others. ### YJIT consideration I consulted @maximecb about this proposal, and according to her, accepting variable arguments makes it harder for YJIT to optimize. I suspect consistency with other APIs trumps the performance consideration, but I think it's worth mentioning. -- https://bugs.ruby-lang.org/
Issue #20594 has been updated by byroot (Jean Boussier).
The name is `String#append_as_bytes`, and the name provides a framing of "reinterpretation" that helps to explain the behavior of the method.
I like that name, I think it clears out any possible confusion.
`append_as_bytes(*objects)`
I retired the `*objects` from my proposal because I've been told it'd be too hard to optimize, but if you believe it's no problem, I definitely prefer it as it's much more Ruby like, and was my initial proposal.
For integers, the i & 0xff masking behavior comes from String#setbyte.
I agree it makes sense to mirror `setbyte` rather than `String#<<(Integer)` here. ---------------------------------------- Feature #20594: A new String method to append bytes while preserving encoding https://bugs.ruby-lang.org/issues/20594#change-109355 * Author: byroot (Jean Boussier) * Status: Open * Assignee: byroot (Jean Boussier) ---------------------------------------- ### Context When working with binary protocols such as `protobuf` or `MessagePack`, you may often need to assemble multiple strings of different encoding: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title << 255 << body.bytesize << body end end Post.new("Hello", "World").serialize("somedata".b) # => "somedata\xFF\x05Hello\xFF\x05World" #<Encoding:ASCII-8BIT> ``` The problem in the above case, is that because `Encoding::ASCII_8BIT` is declared as ASCII compatible, if one of the appended string contains bytes outside the ASCII range, string is automatically promoted to another encoding, which then leads to encoding issues: ```ruby Post.new("H€llo", "Wôrld").serialize("somedata".b) # => incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError) ``` In many cases, you want to append to a String without changing the receiver's encoding. The issue isn't exclusive to binary protocols and formats, it also happen with ASCII protocols that accept arbitrary bytes inline, like Redis's RESP protocol or even HTTP/1.1. ### Previous discussion There was a similar feature request a while ago, but it was abandoned: https://bugs.ruby-lang.org/issues/14975 ### Existing solutions You can of course always cast the strings you append to avoid this problem: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title.b << 255 << body.bytesize << body.b end end ``` But this cause a lot of needless allocations. You'd think you could also use `bytesplice`, but it actually has the same issue: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize buf.bytesplice(buf.bytesize, title.bytesize, title) buf << 255 << body.bytesize buf.bytesplice(buf.bytesize, body.bytesize, title) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => 'String#bytesplice': incompatible character encodings: BINARY (ASCII-8BIT) and UTF-8 (Encoding::CompatibilityError) ``` And even if it worked, it would be very unergonomic. ### Proposal: a `byteconcat` method A solution to this would be to add a new `byteconcat` method, that could be shimed as: ```ruby class String def byteconcat(*strings) strings.map! do |s| if s.is_a?(String) && s.encoding != encoding s.dup.force_encoding(encoding) else s end end concat(*strings) end end Post = Struct.new(:title, :body) do def serialize(buf) buf.byteconcat( 255, title.bytesize, title, 255, body.bytesize, body, ) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => "somedata\xFF\aH\xE2\x82\xACllo\xFF\x06W\xC3\xB4rld" #<Encoding:ASCII-8BIT> ``` But of course a builtin implementation wouldn't need to dup the arguments. Like other `byte*` methods, it's the responsibility of the caller to ensure the resulting string has a valid encoding, or to deal with it if not. ### Method name and signature #### Name This proposal suggests `String#byteconcat`, to mirror `String#concat`, but other names are possible: - `byteappend` (like `Array#append`) - `bytepush` (like `Array#push`) #### Signature This proposal makes `byteconcat` accept either `String` or `Integer` (in char range) arguments like `concat`. I believe it makes sense for consistency and also because it's not uncommon for protocols to have some byte based segments, and Integers are more convenient there. The proposed method also accept variable arguments for consistency with `String#concat`, `Array#push`, `Array#append`. The proposed method returns self, like `concat` and others. ### YJIT consideration I consulted @maximecb about this proposal, and according to her, accepting variable arguments makes it harder for YJIT to optimize. I suspect consistency with other APIs trumps the performance consideration, but I think it's worth mentioning. -- https://bugs.ruby-lang.org/
Issue #20594 has been updated by tenderlovemaking (Aaron Patterson). No opinion on method name / API, but we've verified effectiveness of this optimization using our protobuf implementation and you can see the results [here](https://github.com/Shopify/protoboeuf/pull/116). ---------------------------------------- Feature #20594: A new String method to append bytes while preserving encoding https://bugs.ruby-lang.org/issues/20594#change-109362 * Author: byroot (Jean Boussier) * Status: Open * Assignee: byroot (Jean Boussier) ---------------------------------------- ### Context When working with binary protocols such as `protobuf` or `MessagePack`, you may often need to assemble multiple strings of different encoding: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title << 255 << body.bytesize << body end end Post.new("Hello", "World").serialize("somedata".b) # => "somedata\xFF\x05Hello\xFF\x05World" #<Encoding:ASCII-8BIT> ``` The problem in the above case, is that because `Encoding::ASCII_8BIT` is declared as ASCII compatible, if one of the appended string contains bytes outside the ASCII range, string is automatically promoted to another encoding, which then leads to encoding issues: ```ruby Post.new("H€llo", "Wôrld").serialize("somedata".b) # => incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError) ``` In many cases, you want to append to a String without changing the receiver's encoding. The issue isn't exclusive to binary protocols and formats, it also happen with ASCII protocols that accept arbitrary bytes inline, like Redis's RESP protocol or even HTTP/1.1. ### Previous discussion There was a similar feature request a while ago, but it was abandoned: https://bugs.ruby-lang.org/issues/14975 ### Existing solutions You can of course always cast the strings you append to avoid this problem: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title.b << 255 << body.bytesize << body.b end end ``` But this cause a lot of needless allocations. You'd think you could also use `bytesplice`, but it actually has the same issue: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize buf.bytesplice(buf.bytesize, title.bytesize, title) buf << 255 << body.bytesize buf.bytesplice(buf.bytesize, body.bytesize, title) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => 'String#bytesplice': incompatible character encodings: BINARY (ASCII-8BIT) and UTF-8 (Encoding::CompatibilityError) ``` And even if it worked, it would be very unergonomic. ### Proposal: a `byteconcat` method A solution to this would be to add a new `byteconcat` method, that could be shimed as: ```ruby class String def byteconcat(*strings) strings.map! do |s| if s.is_a?(String) && s.encoding != encoding s.dup.force_encoding(encoding) else s end end concat(*strings) end end Post = Struct.new(:title, :body) do def serialize(buf) buf.byteconcat( 255, title.bytesize, title, 255, body.bytesize, body, ) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => "somedata\xFF\aH\xE2\x82\xACllo\xFF\x06W\xC3\xB4rld" #<Encoding:ASCII-8BIT> ``` But of course a builtin implementation wouldn't need to dup the arguments. Like other `byte*` methods, it's the responsibility of the caller to ensure the resulting string has a valid encoding, or to deal with it if not. ### Method name and signature #### Name This proposal suggests `String#byteconcat`, to mirror `String#concat`, but other names are possible: - `byteappend` (like `Array#append`) - `bytepush` (like `Array#push`) #### Signature This proposal makes `byteconcat` accept either `String` or `Integer` (in char range) arguments like `concat`. I believe it makes sense for consistency and also because it's not uncommon for protocols to have some byte based segments, and Integers are more convenient there. The proposed method also accept variable arguments for consistency with `String#concat`, `Array#push`, `Array#append`. The proposed method returns self, like `concat` and others. ### YJIT consideration I consulted @maximecb about this proposal, and according to her, accepting variable arguments makes it harder for YJIT to optimize. I suspect consistency with other APIs trumps the performance consideration, but I think it's worth mentioning. -- https://bugs.ruby-lang.org/
Issue #20594 has been updated by mame (Yusuke Endoh). Alan's proposal looks good to me. I don't think it is Ruby to design such an ordinary API with extreme care for optimization. In fact, the same design as Alan's proposal was discussed at the dev meeting (sorry it was not written in the meeting log). The one difference between it and Alan's is that an exception should be thrown for Integers other than 0..255 until we have a convincing use case. I personally think it is reasonably convincing that it is the same as `setbyte`, though. ---------------------------------------- Feature #20594: A new String method to append bytes while preserving encoding https://bugs.ruby-lang.org/issues/20594#change-109392 * Author: byroot (Jean Boussier) * Status: Open * Assignee: byroot (Jean Boussier) ---------------------------------------- ### Context When working with binary protocols such as `protobuf` or `MessagePack`, you may often need to assemble multiple strings of different encoding: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title << 255 << body.bytesize << body end end Post.new("Hello", "World").serialize("somedata".b) # => "somedata\xFF\x05Hello\xFF\x05World" #<Encoding:ASCII-8BIT> ``` The problem in the above case, is that because `Encoding::ASCII_8BIT` is declared as ASCII compatible, if one of the appended string contains bytes outside the ASCII range, string is automatically promoted to another encoding, which then leads to encoding issues: ```ruby Post.new("H€llo", "Wôrld").serialize("somedata".b) # => incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError) ``` In many cases, you want to append to a String without changing the receiver's encoding. The issue isn't exclusive to binary protocols and formats, it also happen with ASCII protocols that accept arbitrary bytes inline, like Redis's RESP protocol or even HTTP/1.1. ### Previous discussion There was a similar feature request a while ago, but it was abandoned: https://bugs.ruby-lang.org/issues/14975 ### Existing solutions You can of course always cast the strings you append to avoid this problem: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title.b << 255 << body.bytesize << body.b end end ``` But this cause a lot of needless allocations. You'd think you could also use `bytesplice`, but it actually has the same issue: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize buf.bytesplice(buf.bytesize, title.bytesize, title) buf << 255 << body.bytesize buf.bytesplice(buf.bytesize, body.bytesize, title) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => 'String#bytesplice': incompatible character encodings: BINARY (ASCII-8BIT) and UTF-8 (Encoding::CompatibilityError) ``` And even if it worked, it would be very unergonomic. ### Proposal: a `byteconcat` method A solution to this would be to add a new `byteconcat` method, that could be shimed as: ```ruby class String def byteconcat(*strings) strings.map! do |s| if s.is_a?(String) && s.encoding != encoding s.dup.force_encoding(encoding) else s end end concat(*strings) end end Post = Struct.new(:title, :body) do def serialize(buf) buf.byteconcat( 255, title.bytesize, title, 255, body.bytesize, body, ) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => "somedata\xFF\aH\xE2\x82\xACllo\xFF\x06W\xC3\xB4rld" #<Encoding:ASCII-8BIT> ``` But of course a builtin implementation wouldn't need to dup the arguments. Like other `byte*` methods, it's the responsibility of the caller to ensure the resulting string has a valid encoding, or to deal with it if not. ### Method name and signature #### Name This proposal suggests `String#byteconcat`, to mirror `String#concat`, but other names are possible: - `byteappend` (like `Array#append`) - `bytepush` (like `Array#push`) #### Signature This proposal makes `byteconcat` accept either `String` or `Integer` (in char range) arguments like `concat`. I believe it makes sense for consistency and also because it's not uncommon for protocols to have some byte based segments, and Integers are more convenient there. The proposed method also accept variable arguments for consistency with `String#concat`, `Array#push`, `Array#append`. The proposed method returns self, like `concat` and others. ### YJIT consideration I consulted @maximecb about this proposal, and according to her, accepting variable arguments makes it harder for YJIT to optimize. I suspect consistency with other APIs trumps the performance consideration, but I think it's worth mentioning. -- https://bugs.ruby-lang.org/
Issue #20594 has been updated by matz (Yukihiro Matsumoto). `String#append_as_bytes` looks good to me too. Accepted. Matz. ---------------------------------------- Feature #20594: A new String method to append bytes while preserving encoding https://bugs.ruby-lang.org/issues/20594#change-109630 * Author: byroot (Jean Boussier) * Status: Open * Assignee: byroot (Jean Boussier) ---------------------------------------- ### Context When working with binary protocols such as `protobuf` or `MessagePack`, you may often need to assemble multiple strings of different encoding: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title << 255 << body.bytesize << body end end Post.new("Hello", "World").serialize("somedata".b) # => "somedata\xFF\x05Hello\xFF\x05World" #<Encoding:ASCII-8BIT> ``` The problem in the above case, is that because `Encoding::ASCII_8BIT` is declared as ASCII compatible, if one of the appended string contains bytes outside the ASCII range, string is automatically promoted to another encoding, which then leads to encoding issues: ```ruby Post.new("H€llo", "Wôrld").serialize("somedata".b) # => incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError) ``` In many cases, you want to append to a String without changing the receiver's encoding. The issue isn't exclusive to binary protocols and formats, it also happen with ASCII protocols that accept arbitrary bytes inline, like Redis's RESP protocol or even HTTP/1.1. ### Previous discussion There was a similar feature request a while ago, but it was abandoned: https://bugs.ruby-lang.org/issues/14975 ### Existing solutions You can of course always cast the strings you append to avoid this problem: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title.b << 255 << body.bytesize << body.b end end ``` But this cause a lot of needless allocations. You'd think you could also use `bytesplice`, but it actually has the same issue: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize buf.bytesplice(buf.bytesize, title.bytesize, title) buf << 255 << body.bytesize buf.bytesplice(buf.bytesize, body.bytesize, title) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => 'String#bytesplice': incompatible character encodings: BINARY (ASCII-8BIT) and UTF-8 (Encoding::CompatibilityError) ``` And even if it worked, it would be very unergonomic. ### Proposal: a `byteconcat` method A solution to this would be to add a new `byteconcat` method, that could be shimed as: ```ruby class String def byteconcat(*strings) strings.map! do |s| if s.is_a?(String) && s.encoding != encoding s.dup.force_encoding(encoding) else s end end concat(*strings) end end Post = Struct.new(:title, :body) do def serialize(buf) buf.byteconcat( 255, title.bytesize, title, 255, body.bytesize, body, ) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => "somedata\xFF\aH\xE2\x82\xACllo\xFF\x06W\xC3\xB4rld" #<Encoding:ASCII-8BIT> ``` But of course a builtin implementation wouldn't need to dup the arguments. Like other `byte*` methods, it's the responsibility of the caller to ensure the resulting string has a valid encoding, or to deal with it if not. ### Method name and signature #### Name This proposal suggests `String#byteconcat`, to mirror `String#concat`, but other names are possible: - `byteappend` (like `Array#append`) - `bytepush` (like `Array#push`) #### Signature This proposal makes `byteconcat` accept either `String` or `Integer` (in char range) arguments like `concat`. I believe it makes sense for consistency and also because it's not uncommon for protocols to have some byte based segments, and Integers are more convenient there. The proposed method also accept variable arguments for consistency with `String#concat`, `Array#push`, `Array#append`. The proposed method returns self, like `concat` and others. ### YJIT consideration I consulted @maximecb about this proposal, and according to her, accepting variable arguments makes it harder for YJIT to optimize. I suspect consistency with other APIs trumps the performance consideration, but I think it's worth mentioning. -- https://bugs.ruby-lang.org/
Issue #20594 has been updated by Dan0042 (Daniel DeLorme).
For integers, the i & 0xff masking behavior comes from String#setbyte.
I agree it makes sense to mirror `setbyte` rather than `String#<<(Integer)` here.
Can anyone explain what is the use case for this behavior? It seems strange to me that silently truncating integers would be a desirable behavior. Especially given the example above `append_as_bytes(title.bytesize, title)` would result in corrupted output if title is larger than 255 bytes. ---------------------------------------- Feature #20594: A new String method to append bytes while preserving encoding https://bugs.ruby-lang.org/issues/20594#change-109708 * Author: byroot (Jean Boussier) * Status: Closed * Assignee: byroot (Jean Boussier) ---------------------------------------- ### Context When working with binary protocols such as `protobuf` or `MessagePack`, you may often need to assemble multiple strings of different encoding: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title << 255 << body.bytesize << body end end Post.new("Hello", "World").serialize("somedata".b) # => "somedata\xFF\x05Hello\xFF\x05World" #<Encoding:ASCII-8BIT> ``` The problem in the above case, is that because `Encoding::ASCII_8BIT` is declared as ASCII compatible, if one of the appended string contains bytes outside the ASCII range, string is automatically promoted to another encoding, which then leads to encoding issues: ```ruby Post.new("H€llo", "Wôrld").serialize("somedata".b) # => incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError) ``` In many cases, you want to append to a String without changing the receiver's encoding. The issue isn't exclusive to binary protocols and formats, it also happen with ASCII protocols that accept arbitrary bytes inline, like Redis's RESP protocol or even HTTP/1.1. ### Previous discussion There was a similar feature request a while ago, but it was abandoned: https://bugs.ruby-lang.org/issues/14975 ### Existing solutions You can of course always cast the strings you append to avoid this problem: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title.b << 255 << body.bytesize << body.b end end ``` But this cause a lot of needless allocations. You'd think you could also use `bytesplice`, but it actually has the same issue: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize buf.bytesplice(buf.bytesize, title.bytesize, title) buf << 255 << body.bytesize buf.bytesplice(buf.bytesize, body.bytesize, title) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => 'String#bytesplice': incompatible character encodings: BINARY (ASCII-8BIT) and UTF-8 (Encoding::CompatibilityError) ``` And even if it worked, it would be very unergonomic. ### Proposal: a `byteconcat` method A solution to this would be to add a new `byteconcat` method, that could be shimed as: ```ruby class String def byteconcat(*strings) strings.map! do |s| if s.is_a?(String) && s.encoding != encoding s.dup.force_encoding(encoding) else s end end concat(*strings) end end Post = Struct.new(:title, :body) do def serialize(buf) buf.byteconcat( 255, title.bytesize, title, 255, body.bytesize, body, ) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => "somedata\xFF\aH\xE2\x82\xACllo\xFF\x06W\xC3\xB4rld" #<Encoding:ASCII-8BIT> ``` But of course a builtin implementation wouldn't need to dup the arguments. Like other `byte*` methods, it's the responsibility of the caller to ensure the resulting string has a valid encoding, or to deal with it if not. ### Method name and signature #### Name This proposal suggests `String#byteconcat`, to mirror `String#concat`, but other names are possible: - `byteappend` (like `Array#append`) - `bytepush` (like `Array#push`) #### Signature This proposal makes `byteconcat` accept either `String` or `Integer` (in char range) arguments like `concat`. I believe it makes sense for consistency and also because it's not uncommon for protocols to have some byte based segments, and Integers are more convenient there. The proposed method also accept variable arguments for consistency with `String#concat`, `Array#push`, `Array#append`. The proposed method returns self, like `concat` and others. ### YJIT consideration I consulted @maximecb about this proposal, and according to her, accepting variable arguments makes it harder for YJIT to optimize. I suspect consistency with other APIs trumps the performance consideration, but I think it's worth mentioning. -- https://bugs.ruby-lang.org/
Issue #20594 has been updated by tenderlovemaking (Aaron Patterson). Dan0042 (Daniel DeLorme) wrote in #note-27:
For integers, the i & 0xff masking behavior comes from String#setbyte.
I agree it makes sense to mirror `setbyte` rather than `String#<<(Integer)` here.
Can anyone explain what is the use case for this behavior? It seems strange to me that silently truncating integers would be a desirable behavior. Especially given the example above `append_as_bytes(title.bytesize, title)` would result in corrupted output if title is larger than 255 bytes.
I don't think there is a specific use case (or maybe the use case is the same as when providing out-of-bounds numbers to `setbyte`?). To me the question is "what to do with out-of-bounds input?", and it seems `setbyte` will just truncate bad input so `append_as_bytes` is doing the same. ---------------------------------------- Feature #20594: A new String method to append bytes while preserving encoding https://bugs.ruby-lang.org/issues/20594#change-109709 * Author: byroot (Jean Boussier) * Status: Closed * Assignee: byroot (Jean Boussier) ---------------------------------------- ### Context When working with binary protocols such as `protobuf` or `MessagePack`, you may often need to assemble multiple strings of different encoding: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title << 255 << body.bytesize << body end end Post.new("Hello", "World").serialize("somedata".b) # => "somedata\xFF\x05Hello\xFF\x05World" #<Encoding:ASCII-8BIT> ``` The problem in the above case, is that because `Encoding::ASCII_8BIT` is declared as ASCII compatible, if one of the appended string contains bytes outside the ASCII range, string is automatically promoted to another encoding, which then leads to encoding issues: ```ruby Post.new("H€llo", "Wôrld").serialize("somedata".b) # => incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError) ``` In many cases, you want to append to a String without changing the receiver's encoding. The issue isn't exclusive to binary protocols and formats, it also happen with ASCII protocols that accept arbitrary bytes inline, like Redis's RESP protocol or even HTTP/1.1. ### Previous discussion There was a similar feature request a while ago, but it was abandoned: https://bugs.ruby-lang.org/issues/14975 ### Existing solutions You can of course always cast the strings you append to avoid this problem: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title.b << 255 << body.bytesize << body.b end end ``` But this cause a lot of needless allocations. You'd think you could also use `bytesplice`, but it actually has the same issue: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize buf.bytesplice(buf.bytesize, title.bytesize, title) buf << 255 << body.bytesize buf.bytesplice(buf.bytesize, body.bytesize, title) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => 'String#bytesplice': incompatible character encodings: BINARY (ASCII-8BIT) and UTF-8 (Encoding::CompatibilityError) ``` And even if it worked, it would be very unergonomic. ### Proposal: a `byteconcat` method A solution to this would be to add a new `byteconcat` method, that could be shimed as: ```ruby class String def byteconcat(*strings) strings.map! do |s| if s.is_a?(String) && s.encoding != encoding s.dup.force_encoding(encoding) else s end end concat(*strings) end end Post = Struct.new(:title, :body) do def serialize(buf) buf.byteconcat( 255, title.bytesize, title, 255, body.bytesize, body, ) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => "somedata\xFF\aH\xE2\x82\xACllo\xFF\x06W\xC3\xB4rld" #<Encoding:ASCII-8BIT> ``` But of course a builtin implementation wouldn't need to dup the arguments. Like other `byte*` methods, it's the responsibility of the caller to ensure the resulting string has a valid encoding, or to deal with it if not. ### Method name and signature #### Name This proposal suggests `String#byteconcat`, to mirror `String#concat`, but other names are possible: - `byteappend` (like `Array#append`) - `bytepush` (like `Array#push`) #### Signature This proposal makes `byteconcat` accept either `String` or `Integer` (in char range) arguments like `concat`. I believe it makes sense for consistency and also because it's not uncommon for protocols to have some byte based segments, and Integers are more convenient there. The proposed method also accept variable arguments for consistency with `String#concat`, `Array#push`, `Array#append`. The proposed method returns self, like `concat` and others. ### YJIT consideration I consulted @maximecb about this proposal, and according to her, accepting variable arguments makes it harder for YJIT to optimize. I suspect consistency with other APIs trumps the performance consideration, but I think it's worth mentioning. -- https://bugs.ruby-lang.org/
Issue #20594 has been updated by Dan0042 (Daniel DeLorme). I understand the idea of keeping things consistent. But why does `setbyte` have this behavior in the first place? A few years back there was an attempt to check bounds, and then in #15460 Matz apparently decided he "prefers mod 256 behaviour for larger numbers". But there's no context or explanation for this preference, so I'm left mystified. ---------------------------------------- Feature #20594: A new String method to append bytes while preserving encoding https://bugs.ruby-lang.org/issues/20594#change-109711 * Author: byroot (Jean Boussier) * Status: Closed * Assignee: byroot (Jean Boussier) ---------------------------------------- ### Context When working with binary protocols such as `protobuf` or `MessagePack`, you may often need to assemble multiple strings of different encoding: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title << 255 << body.bytesize << body end end Post.new("Hello", "World").serialize("somedata".b) # => "somedata\xFF\x05Hello\xFF\x05World" #<Encoding:ASCII-8BIT> ``` The problem in the above case, is that because `Encoding::ASCII_8BIT` is declared as ASCII compatible, if one of the appended string contains bytes outside the ASCII range, string is automatically promoted to another encoding, which then leads to encoding issues: ```ruby Post.new("H€llo", "Wôrld").serialize("somedata".b) # => incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError) ``` In many cases, you want to append to a String without changing the receiver's encoding. The issue isn't exclusive to binary protocols and formats, it also happen with ASCII protocols that accept arbitrary bytes inline, like Redis's RESP protocol or even HTTP/1.1. ### Previous discussion There was a similar feature request a while ago, but it was abandoned: https://bugs.ruby-lang.org/issues/14975 ### Existing solutions You can of course always cast the strings you append to avoid this problem: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title.b << 255 << body.bytesize << body.b end end ``` But this cause a lot of needless allocations. You'd think you could also use `bytesplice`, but it actually has the same issue: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize buf.bytesplice(buf.bytesize, title.bytesize, title) buf << 255 << body.bytesize buf.bytesplice(buf.bytesize, body.bytesize, title) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => 'String#bytesplice': incompatible character encodings: BINARY (ASCII-8BIT) and UTF-8 (Encoding::CompatibilityError) ``` And even if it worked, it would be very unergonomic. ### Proposal: a `byteconcat` method A solution to this would be to add a new `byteconcat` method, that could be shimed as: ```ruby class String def byteconcat(*strings) strings.map! do |s| if s.is_a?(String) && s.encoding != encoding s.dup.force_encoding(encoding) else s end end concat(*strings) end end Post = Struct.new(:title, :body) do def serialize(buf) buf.byteconcat( 255, title.bytesize, title, 255, body.bytesize, body, ) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => "somedata\xFF\aH\xE2\x82\xACllo\xFF\x06W\xC3\xB4rld" #<Encoding:ASCII-8BIT> ``` But of course a builtin implementation wouldn't need to dup the arguments. Like other `byte*` methods, it's the responsibility of the caller to ensure the resulting string has a valid encoding, or to deal with it if not. ### Method name and signature #### Name This proposal suggests `String#byteconcat`, to mirror `String#concat`, but other names are possible: - `byteappend` (like `Array#append`) - `bytepush` (like `Array#push`) #### Signature This proposal makes `byteconcat` accept either `String` or `Integer` (in char range) arguments like `concat`. I believe it makes sense for consistency and also because it's not uncommon for protocols to have some byte based segments, and Integers are more convenient there. The proposed method also accept variable arguments for consistency with `String#concat`, `Array#push`, `Array#append`. The proposed method returns self, like `concat` and others. ### YJIT consideration I consulted @maximecb about this proposal, and according to her, accepting variable arguments makes it harder for YJIT to optimize. I suspect consistency with other APIs trumps the performance consideration, but I think it's worth mentioning. -- https://bugs.ruby-lang.org/
Issue #20594 has been updated by shyouhei (Shyouhei Urabe). Dan0042 (Daniel DeLorme) wrote in #note-29:
I understand the idea of keeping things consistent. But why does `setbyte` have this behavior in the first place? A few years back there was an attempt to check bounds, and then in #15460 Matz apparently decided he "prefers mod 256 behaviour for larger numbers". But there's no context or explanation for this preference, so I'm left mystified.
I remember that in #15460 @gettalong showed us a use case in his HexaPDF when modulo behaviour is in fact useful. Matz was persuaded by that. ---------------------------------------- Feature #20594: A new String method to append bytes while preserving encoding https://bugs.ruby-lang.org/issues/20594#change-109712 * Author: byroot (Jean Boussier) * Status: Closed * Assignee: byroot (Jean Boussier) ---------------------------------------- ### Context When working with binary protocols such as `protobuf` or `MessagePack`, you may often need to assemble multiple strings of different encoding: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title << 255 << body.bytesize << body end end Post.new("Hello", "World").serialize("somedata".b) # => "somedata\xFF\x05Hello\xFF\x05World" #<Encoding:ASCII-8BIT> ``` The problem in the above case, is that because `Encoding::ASCII_8BIT` is declared as ASCII compatible, if one of the appended string contains bytes outside the ASCII range, string is automatically promoted to another encoding, which then leads to encoding issues: ```ruby Post.new("H€llo", "Wôrld").serialize("somedata".b) # => incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError) ``` In many cases, you want to append to a String without changing the receiver's encoding. The issue isn't exclusive to binary protocols and formats, it also happen with ASCII protocols that accept arbitrary bytes inline, like Redis's RESP protocol or even HTTP/1.1. ### Previous discussion There was a similar feature request a while ago, but it was abandoned: https://bugs.ruby-lang.org/issues/14975 ### Existing solutions You can of course always cast the strings you append to avoid this problem: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title.b << 255 << body.bytesize << body.b end end ``` But this cause a lot of needless allocations. You'd think you could also use `bytesplice`, but it actually has the same issue: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize buf.bytesplice(buf.bytesize, title.bytesize, title) buf << 255 << body.bytesize buf.bytesplice(buf.bytesize, body.bytesize, title) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => 'String#bytesplice': incompatible character encodings: BINARY (ASCII-8BIT) and UTF-8 (Encoding::CompatibilityError) ``` And even if it worked, it would be very unergonomic. ### Proposal: a `byteconcat` method A solution to this would be to add a new `byteconcat` method, that could be shimed as: ```ruby class String def byteconcat(*strings) strings.map! do |s| if s.is_a?(String) && s.encoding != encoding s.dup.force_encoding(encoding) else s end end concat(*strings) end end Post = Struct.new(:title, :body) do def serialize(buf) buf.byteconcat( 255, title.bytesize, title, 255, body.bytesize, body, ) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => "somedata\xFF\aH\xE2\x82\xACllo\xFF\x06W\xC3\xB4rld" #<Encoding:ASCII-8BIT> ``` But of course a builtin implementation wouldn't need to dup the arguments. Like other `byte*` methods, it's the responsibility of the caller to ensure the resulting string has a valid encoding, or to deal with it if not. ### Method name and signature #### Name This proposal suggests `String#byteconcat`, to mirror `String#concat`, but other names are possible: - `byteappend` (like `Array#append`) - `bytepush` (like `Array#push`) #### Signature This proposal makes `byteconcat` accept either `String` or `Integer` (in char range) arguments like `concat`. I believe it makes sense for consistency and also because it's not uncommon for protocols to have some byte based segments, and Integers are more convenient there. The proposed method also accept variable arguments for consistency with `String#concat`, `Array#push`, `Array#append`. The proposed method returns self, like `concat` and others. ### YJIT consideration I consulted @maximecb about this proposal, and according to her, accepting variable arguments makes it harder for YJIT to optimize. I suspect consistency with other APIs trumps the performance consideration, but I think it's worth mentioning. -- https://bugs.ruby-lang.org/
Issue #20594 has been updated by Dan0042 (Daniel DeLorme). Ironically, HexaPDF now uses an explicit modulo 256, so it would actually be fine if #setbyte raised an out-of-bound error. https://github.com/gettalong/hexapdf/blob/master/lib/hexapdf/filter/predicto... imho better safe than sorry. It's very easy to add a modulo when needed, so the built-in modulo 256 has limited usefulness. The opposite, adding a range check in ruby, is much more cumbersome. Just my 2¢ ---------------------------------------- Feature #20594: A new String method to append bytes while preserving encoding https://bugs.ruby-lang.org/issues/20594#change-109713 * Author: byroot (Jean Boussier) * Status: Closed * Assignee: byroot (Jean Boussier) ---------------------------------------- ### Context When working with binary protocols such as `protobuf` or `MessagePack`, you may often need to assemble multiple strings of different encoding: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title << 255 << body.bytesize << body end end Post.new("Hello", "World").serialize("somedata".b) # => "somedata\xFF\x05Hello\xFF\x05World" #<Encoding:ASCII-8BIT> ``` The problem in the above case, is that because `Encoding::ASCII_8BIT` is declared as ASCII compatible, if one of the appended string contains bytes outside the ASCII range, string is automatically promoted to another encoding, which then leads to encoding issues: ```ruby Post.new("H€llo", "Wôrld").serialize("somedata".b) # => incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError) ``` In many cases, you want to append to a String without changing the receiver's encoding. The issue isn't exclusive to binary protocols and formats, it also happen with ASCII protocols that accept arbitrary bytes inline, like Redis's RESP protocol or even HTTP/1.1. ### Previous discussion There was a similar feature request a while ago, but it was abandoned: https://bugs.ruby-lang.org/issues/14975 ### Existing solutions You can of course always cast the strings you append to avoid this problem: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title.b << 255 << body.bytesize << body.b end end ``` But this cause a lot of needless allocations. You'd think you could also use `bytesplice`, but it actually has the same issue: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize buf.bytesplice(buf.bytesize, title.bytesize, title) buf << 255 << body.bytesize buf.bytesplice(buf.bytesize, body.bytesize, title) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => 'String#bytesplice': incompatible character encodings: BINARY (ASCII-8BIT) and UTF-8 (Encoding::CompatibilityError) ``` And even if it worked, it would be very unergonomic. ### Proposal: a `byteconcat` method A solution to this would be to add a new `byteconcat` method, that could be shimed as: ```ruby class String def byteconcat(*strings) strings.map! do |s| if s.is_a?(String) && s.encoding != encoding s.dup.force_encoding(encoding) else s end end concat(*strings) end end Post = Struct.new(:title, :body) do def serialize(buf) buf.byteconcat( 255, title.bytesize, title, 255, body.bytesize, body, ) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => "somedata\xFF\aH\xE2\x82\xACllo\xFF\x06W\xC3\xB4rld" #<Encoding:ASCII-8BIT> ``` But of course a builtin implementation wouldn't need to dup the arguments. Like other `byte*` methods, it's the responsibility of the caller to ensure the resulting string has a valid encoding, or to deal with it if not. ### Method name and signature #### Name This proposal suggests `String#byteconcat`, to mirror `String#concat`, but other names are possible: - `byteappend` (like `Array#append`) - `bytepush` (like `Array#push`) #### Signature This proposal makes `byteconcat` accept either `String` or `Integer` (in char range) arguments like `concat`. I believe it makes sense for consistency and also because it's not uncommon for protocols to have some byte based segments, and Integers are more convenient there. The proposed method also accept variable arguments for consistency with `String#concat`, `Array#push`, `Array#append`. The proposed method returns self, like `concat` and others. ### YJIT consideration I consulted @maximecb about this proposal, and according to her, accepting variable arguments makes it harder for YJIT to optimize. I suspect consistency with other APIs trumps the performance consideration, but I think it's worth mentioning. -- https://bugs.ruby-lang.org/
Issue #20594 has been updated by gettalong (Thomas Leitner). Dan0042 (Daniel DeLorme) wrote in #note-31:
Ironically, HexaPDF now uses an explicit modulo 256, so it would actually be fine if #setbyte raised an out-of-bound error. https://github.com/gettalong/hexapdf/blob/master/lib/hexapdf/filter/predicto...
Yes, as mentioned in the linked issue #15460 I implemented this so that HexaPDF remained compatible with 2.6.0. Since I dropped support for Ruby 2.6 in HexaPDF at the start of 2024, I could now change it back again. I don't have a strong opinion on whether `setbyte` or now `append_as_bytes` should do modulo 256 or bounds checking, I was more concerned with compatibility to previous versions back then. ---------------------------------------- Feature #20594: A new String method to append bytes while preserving encoding https://bugs.ruby-lang.org/issues/20594#change-109714 * Author: byroot (Jean Boussier) * Status: Closed * Assignee: byroot (Jean Boussier) ---------------------------------------- ### Context When working with binary protocols such as `protobuf` or `MessagePack`, you may often need to assemble multiple strings of different encoding: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title << 255 << body.bytesize << body end end Post.new("Hello", "World").serialize("somedata".b) # => "somedata\xFF\x05Hello\xFF\x05World" #<Encoding:ASCII-8BIT> ``` The problem in the above case, is that because `Encoding::ASCII_8BIT` is declared as ASCII compatible, if one of the appended string contains bytes outside the ASCII range, string is automatically promoted to another encoding, which then leads to encoding issues: ```ruby Post.new("H€llo", "Wôrld").serialize("somedata".b) # => incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError) ``` In many cases, you want to append to a String without changing the receiver's encoding. The issue isn't exclusive to binary protocols and formats, it also happen with ASCII protocols that accept arbitrary bytes inline, like Redis's RESP protocol or even HTTP/1.1. ### Previous discussion There was a similar feature request a while ago, but it was abandoned: https://bugs.ruby-lang.org/issues/14975 ### Existing solutions You can of course always cast the strings you append to avoid this problem: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title.b << 255 << body.bytesize << body.b end end ``` But this cause a lot of needless allocations. You'd think you could also use `bytesplice`, but it actually has the same issue: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize buf.bytesplice(buf.bytesize, title.bytesize, title) buf << 255 << body.bytesize buf.bytesplice(buf.bytesize, body.bytesize, title) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => 'String#bytesplice': incompatible character encodings: BINARY (ASCII-8BIT) and UTF-8 (Encoding::CompatibilityError) ``` And even if it worked, it would be very unergonomic. ### Proposal: a `byteconcat` method A solution to this would be to add a new `byteconcat` method, that could be shimed as: ```ruby class String def byteconcat(*strings) strings.map! do |s| if s.is_a?(String) && s.encoding != encoding s.dup.force_encoding(encoding) else s end end concat(*strings) end end Post = Struct.new(:title, :body) do def serialize(buf) buf.byteconcat( 255, title.bytesize, title, 255, body.bytesize, body, ) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => "somedata\xFF\aH\xE2\x82\xACllo\xFF\x06W\xC3\xB4rld" #<Encoding:ASCII-8BIT> ``` But of course a builtin implementation wouldn't need to dup the arguments. Like other `byte*` methods, it's the responsibility of the caller to ensure the resulting string has a valid encoding, or to deal with it if not. ### Method name and signature #### Name This proposal suggests `String#byteconcat`, to mirror `String#concat`, but other names are possible: - `byteappend` (like `Array#append`) - `bytepush` (like `Array#push`) #### Signature This proposal makes `byteconcat` accept either `String` or `Integer` (in char range) arguments like `concat`. I believe it makes sense for consistency and also because it's not uncommon for protocols to have some byte based segments, and Integers are more convenient there. The proposed method also accept variable arguments for consistency with `String#concat`, `Array#push`, `Array#append`. The proposed method returns self, like `concat` and others. ### YJIT consideration I consulted @maximecb about this proposal, and according to her, accepting variable arguments makes it harder for YJIT to optimize. I suspect consistency with other APIs trumps the performance consideration, but I think it's worth mentioning. -- https://bugs.ruby-lang.org/
Issue #20594 has been updated by Eregon (Benoit Daloze). I agree with @Dan0042, the new API should not have this error-prone behavior, it should raise if out of bounds instead of silent truncation. I also think we should fix setbyte to not have this behavior, but that's a different issue. There it might be slightly less error prone as `setbyte` is clearly meant to write a single byte. Still, silent data truncation is always a bad idea. ---------------------------------------- Feature #20594: A new String method to append bytes while preserving encoding https://bugs.ruby-lang.org/issues/20594#change-109715 * Author: byroot (Jean Boussier) * Status: Closed * Assignee: byroot (Jean Boussier) ---------------------------------------- ### Context When working with binary protocols such as `protobuf` or `MessagePack`, you may often need to assemble multiple strings of different encoding: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title << 255 << body.bytesize << body end end Post.new("Hello", "World").serialize("somedata".b) # => "somedata\xFF\x05Hello\xFF\x05World" #<Encoding:ASCII-8BIT> ``` The problem in the above case, is that because `Encoding::ASCII_8BIT` is declared as ASCII compatible, if one of the appended string contains bytes outside the ASCII range, string is automatically promoted to another encoding, which then leads to encoding issues: ```ruby Post.new("H€llo", "Wôrld").serialize("somedata".b) # => incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError) ``` In many cases, you want to append to a String without changing the receiver's encoding. The issue isn't exclusive to binary protocols and formats, it also happen with ASCII protocols that accept arbitrary bytes inline, like Redis's RESP protocol or even HTTP/1.1. ### Previous discussion There was a similar feature request a while ago, but it was abandoned: https://bugs.ruby-lang.org/issues/14975 ### Existing solutions You can of course always cast the strings you append to avoid this problem: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title.b << 255 << body.bytesize << body.b end end ``` But this cause a lot of needless allocations. You'd think you could also use `bytesplice`, but it actually has the same issue: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize buf.bytesplice(buf.bytesize, title.bytesize, title) buf << 255 << body.bytesize buf.bytesplice(buf.bytesize, body.bytesize, title) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => 'String#bytesplice': incompatible character encodings: BINARY (ASCII-8BIT) and UTF-8 (Encoding::CompatibilityError) ``` And even if it worked, it would be very unergonomic. ### Proposal: a `byteconcat` method A solution to this would be to add a new `byteconcat` method, that could be shimed as: ```ruby class String def byteconcat(*strings) strings.map! do |s| if s.is_a?(String) && s.encoding != encoding s.dup.force_encoding(encoding) else s end end concat(*strings) end end Post = Struct.new(:title, :body) do def serialize(buf) buf.byteconcat( 255, title.bytesize, title, 255, body.bytesize, body, ) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => "somedata\xFF\aH\xE2\x82\xACllo\xFF\x06W\xC3\xB4rld" #<Encoding:ASCII-8BIT> ``` But of course a builtin implementation wouldn't need to dup the arguments. Like other `byte*` methods, it's the responsibility of the caller to ensure the resulting string has a valid encoding, or to deal with it if not. ### Method name and signature #### Name This proposal suggests `String#byteconcat`, to mirror `String#concat`, but other names are possible: - `byteappend` (like `Array#append`) - `bytepush` (like `Array#push`) #### Signature This proposal makes `byteconcat` accept either `String` or `Integer` (in char range) arguments like `concat`. I believe it makes sense for consistency and also because it's not uncommon for protocols to have some byte based segments, and Integers are more convenient there. The proposed method also accept variable arguments for consistency with `String#concat`, `Array#push`, `Array#append`. The proposed method returns self, like `concat` and others. ### YJIT consideration I consulted @maximecb about this proposal, and according to her, accepting variable arguments makes it harder for YJIT to optimize. I suspect consistency with other APIs trumps the performance consideration, but I think it's worth mentioning. -- https://bugs.ruby-lang.org/
participants (12)
-
alanwu (Alan Wu) -
byroot (Jean Boussier)
-
Dan0042 (Daniel DeLorme)
-
duerst
-
Eregon (Benoit Daloze)
-
gettalong (Thomas Leitner)
-
mame (Yusuke Endoh)
-
matz (Yukihiro Matsumoto)
-
maximecb (Maxime Chevalier-Boisvert)
-
shugo (Shugo Maeda)
-
shyouhei (Shyouhei Urabe)
-
tenderlovemaking (Aaron Patterson)