[ruby-core:119119] [Ruby master Feature#20594] A new String method to append bytes while preserving encoding

11 Sep 2024

      Issue #20594 has been updated by shyouhei (Shyouhei Urabe).

Dan0042 (Daniel DeLorme) wrote in #note-29:
...
I understand the idea of keeping things consistent. But why does `setbyte` have this behavior in the first place? A few years back there was an attempt to check bounds, and then in #15460 Matz apparently decided he "prefers mod 256 behaviour for larger numbers". But there's no context or explanation for this preference, so I'm left mystified.
I remember that in #15460 @gettalong showed us a use case in his HexaPDF when modulo behaviour is in fact useful.  Matz was persuaded by that.

----------------------------------------
Feature #20594: A new String method to append bytes while preserving encoding
https://bugs.ruby-lang.org/issues/20594#change-109712

* Author: byroot (Jean Boussier)
* Status: Closed
* Assignee: byroot (Jean Boussier)
----------------------------------------
### Context

When working with binary protocols such as `protobuf` or `MessagePack`, you may often need to assemble multiple
strings of different encoding:

```ruby
Post = Struct.new(:title, :body) do
  def serialize(buf)
    buf <<
      255 << title.bytesize << title <<
      255 << body.bytesize << body
  end
end

Post.new("Hello", "World").serialize("somedata".b) # => "somedata\xFF\x05Hello\xFF\x05World" #<Encoding:ASCII-8BIT>
```

The problem in the above case, is that because `Encoding::ASCII_8BIT` is declared as ASCII compatible,
if one of the appended string contains bytes outside the ASCII range, string is automatically promoted
to another encoding, which then leads to encoding issues:

```ruby
Post.new("H€llo", "Wôrld").serialize("somedata".b) # => incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError)
```

In many cases, you want to append to a String without changing the receiver's encoding.

The issue isn't exclusive to binary protocols and formats, it also happen with ASCII protocols that accept arbitrary bytes inline,
like Redis's RESP protocol or even HTTP/1.1.

### Previous discussion

There was a similar feature request a while ago, but it was abandoned: https://bugs.ruby-lang.org/issues/14975

### Existing solutions

You can of course always cast the strings you append to avoid this problem:

```ruby
Post = Struct.new(:title, :body) do
  def serialize(buf)
    buf <<
      255 << title.bytesize << title.b <<
      255 << body.bytesize << body.b
  end
end
```

But this cause a lot of needless allocations.

You'd think you could also use `bytesplice`, but it actually has the same issue:

```ruby
Post = Struct.new(:title, :body) do
  def serialize(buf)
    buf << 255 << title.bytesize
    buf.bytesplice(buf.bytesize, title.bytesize, title)
    buf << 255 << body.bytesize
    buf.bytesplice(buf.bytesize, body.bytesize, title)
  end
end
Post.new("H€llo", "Wôrld").serialize("somedata".b) # => 'String#bytesplice': incompatible character encodings: BINARY (ASCII-8BIT) and UTF-8 (Encoding::CompatibilityError)
```

And even if it worked, it would be very unergonomic.

### Proposal: a `byteconcat` method

A solution to this would be to add a new `byteconcat` method, that could be shimed as:

```ruby
class String
  def byteconcat(*strings)
    strings.map! do |s|
      if s.is_a?(String) && s.encoding != encoding
        s.dup.force_encoding(encoding)
      else
        s
      end
    end
    concat(*strings)
  end
end

Post = Struct.new(:title, :body) do
  def serialize(buf)
    buf.byteconcat(
      255, title.bytesize, title,
      255, body.bytesize, body,
    )
  end
end

Post.new("H€llo", "Wôrld").serialize("somedata".b) # => "somedata\xFF\aH\xE2\x82\xACllo\xFF\x06W\xC3\xB4rld" #<Encoding:ASCII-8BIT>
```

But of course a builtin implementation wouldn't need to dup the arguments.

Like other `byte*` methods, it's the responsibility of the caller to ensure the resulting string has a valid encoding, or
to deal with it if not.

### Method name and signature

#### Name

This proposal suggests `String#byteconcat`, to mirror `String#concat`, but other names are possible:

  - `byteappend` (like `Array#append`)
  - `bytepush`  (like `Array#push`)

#### Signature

This proposal makes `byteconcat` accept either `String` or `Integer` (in char range) arguments like `concat`. I believe it makes sense for consistency and also because it's not uncommon for protocols to have some byte based segments, and Integers are more convenient there.

The proposed method also accept variable arguments for consistency with `String#concat`, `Array#push`, `Array#append`.

The proposed method returns self, like `concat` and others.

### YJIT consideration

I consulted @maximecb about this proposal, and according to her, accepting variable arguments makes it harder for YJIT to optimize.
I suspect consistency with other APIs trumps the performance consideration, but I think it's worth mentioning.

-- 
https://bugs.ruby-lang.org/