[ruby-core:119122] [Ruby master Feature#20594] A new String method to append bytes while preserving encoding

11 Sep 2024

      Issue #20594 has been updated by gettalong (Thomas Leitner).

Dan0042 (Daniel DeLorme) wrote in #note-31:
...
Ironically, HexaPDF now uses an explicit modulo 256, so it would actually be fine if #setbyte raised an out-of-bound error.
https://github.com/gettalong/hexapdf/blob/master/lib/hexapdf/filter/predicto...
Yes, as mentioned in the linked issue #15460 I implemented this so that HexaPDF remained compatible with 2.6.0. Since I dropped support for Ruby 2.6 in HexaPDF at the start of 2024, I could now change it back again.

I don't have a strong opinion on whether `setbyte` or now `append_as_bytes` should do modulo 256 or bounds checking, I was more concerned with compatibility to previous versions back then.

----------------------------------------
Feature #20594: A new String method to append bytes while preserving encoding
https://bugs.ruby-lang.org/issues/20594#change-109714

* Author: byroot (Jean Boussier)
* Status: Closed
* Assignee: byroot (Jean Boussier)
----------------------------------------
### Context

When working with binary protocols such as `protobuf` or `MessagePack`, you may often need to assemble multiple
strings of different encoding:

```ruby
Post = Struct.new(:title, :body) do
  def serialize(buf)
    buf <<
      255 << title.bytesize << title <<
      255 << body.bytesize << body
  end
end

Post.new("Hello", "World").serialize("somedata".b) # => "somedata\xFF\x05Hello\xFF\x05World" #<Encoding:ASCII-8BIT>
```

The problem in the above case, is that because `Encoding::ASCII_8BIT` is declared as ASCII compatible,
if one of the appended string contains bytes outside the ASCII range, string is automatically promoted
to another encoding, which then leads to encoding issues:

```ruby
Post.new("H€llo", "Wôrld").serialize("somedata".b) # => incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError)
```

In many cases, you want to append to a String without changing the receiver's encoding.

The issue isn't exclusive to binary protocols and formats, it also happen with ASCII protocols that accept arbitrary bytes inline,
like Redis's RESP protocol or even HTTP/1.1.

### Previous discussion

There was a similar feature request a while ago, but it was abandoned: https://bugs.ruby-lang.org/issues/14975

### Existing solutions

You can of course always cast the strings you append to avoid this problem:

```ruby
Post = Struct.new(:title, :body) do
  def serialize(buf)
    buf <<
      255 << title.bytesize << title.b <<
      255 << body.bytesize << body.b
  end
end
```

But this cause a lot of needless allocations.

You'd think you could also use `bytesplice`, but it actually has the same issue:

```ruby
Post = Struct.new(:title, :body) do
  def serialize(buf)
    buf << 255 << title.bytesize
    buf.bytesplice(buf.bytesize, title.bytesize, title)
    buf << 255 << body.bytesize
    buf.bytesplice(buf.bytesize, body.bytesize, title)
  end
end
Post.new("H€llo", "Wôrld").serialize("somedata".b) # => 'String#bytesplice': incompatible character encodings: BINARY (ASCII-8BIT) and UTF-8 (Encoding::CompatibilityError)
```

And even if it worked, it would be very unergonomic.

### Proposal: a `byteconcat` method

A solution to this would be to add a new `byteconcat` method, that could be shimed as:

```ruby
class String
  def byteconcat(*strings)
    strings.map! do |s|
      if s.is_a?(String) && s.encoding != encoding
        s.dup.force_encoding(encoding)
      else
        s
      end
    end
    concat(*strings)
  end
end

Post = Struct.new(:title, :body) do
  def serialize(buf)
    buf.byteconcat(
      255, title.bytesize, title,
      255, body.bytesize, body,
    )
  end
end

Post.new("H€llo", "Wôrld").serialize("somedata".b) # => "somedata\xFF\aH\xE2\x82\xACllo\xFF\x06W\xC3\xB4rld" #<Encoding:ASCII-8BIT>
```

But of course a builtin implementation wouldn't need to dup the arguments.

Like other `byte*` methods, it's the responsibility of the caller to ensure the resulting string has a valid encoding, or
to deal with it if not.

### Method name and signature

#### Name

This proposal suggests `String#byteconcat`, to mirror `String#concat`, but other names are possible:

  - `byteappend` (like `Array#append`)
  - `bytepush`  (like `Array#push`)

#### Signature

This proposal makes `byteconcat` accept either `String` or `Integer` (in char range) arguments like `concat`. I believe it makes sense for consistency and also because it's not uncommon for protocols to have some byte based segments, and Integers are more convenient there.

The proposed method also accept variable arguments for consistency with `String#concat`, `Array#push`, `Array#append`.

The proposed method returns self, like `concat` and others.

### YJIT consideration

I consulted @maximecb about this proposal, and according to her, accepting variable arguments makes it harder for YJIT to optimize.
I suspect consistency with other APIs trumps the performance consideration, but I think it's worth mentioning.

-- 
https://bugs.ruby-lang.org/