[ruby-core:118576] [Ruby master Feature#20594] A new String method to append bytes while preserving encoding

11 Jul 2024

      Issue #20594 has been updated by shugo (Shugo Maeda).

Eregon (Benoit Daloze) wrote in #note-11:
...
I wonder if that's a bug of `bytesplice`.
It's not like e.g. `setbyte` would respect the encoding (it cannot possibly).
OTOH `byteindex` also takes a String argument and raises `Encoding::CompatibilityError` for `"é".b.byteindex "é"` so I guess it's not so clear-cut currently (I get the replies above a bit better).
It's intended behavior.
Having a long history of suffering from character encoding issues such as mojibake, we Japanese believe that operations on strings with different encodings should be approached with caution.

`byte-` methods could be considered exceptions, but at present, they are not.  That's why Matz doesn't like the name `byteconcat`.

----------------------------------------
Feature #20594: A new String method to append bytes while preserving encoding
https://bugs.ruby-lang.org/issues/20594#change-109101

* Author: byroot (Jean Boussier)
* Status: Open
* Assignee: byroot (Jean Boussier)
----------------------------------------
### Context

When working with binary protocols such as `protobuf` or `MessagePack`, you may often need to assemble multiple
strings of different encoding:

```ruby
Post = Struct.new(:title, :body) do
  def serialize(buf)
    buf <<
      255 << title.bytesize << title <<
      255 << body.bytesize << body
  end
end

Post.new("Hello", "World").serialize("somedata".b) # => "somedata\xFF\x05Hello\xFF\x05World" #<Encoding:ASCII-8BIT>
```

The problem in the above case, is that because `Encoding::ASCII_8BIT` is declared as ASCII compatible,
if one of the appended string contains bytes outside the ASCII range, string is automatically promoted
to another encoding, which then leads to encoding issues:

```ruby
Post.new("H€llo", "Wôrld").serialize("somedata".b) # => incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError)
```

In many cases, you want to append to a String without changing the receiver's encoding.

The issue isn't exclusive to binary protocols and formats, it also happen with ASCII protocols that accept arbitrary bytes inline,
like Redis's RESP protocol or even HTTP/1.1.

### Previous discussion

There was a similar feature request a while ago, but it was abandoned: https://bugs.ruby-lang.org/issues/14975

### Existing solutions

You can of course always cast the strings you append to avoid this problem:

```ruby
Post = Struct.new(:title, :body) do
  def serialize(buf)
    buf <<
      255 << title.bytesize << title.b <<
      255 << body.bytesize << body.b
  end
end
```

But this cause a lot of needless allocations.

You'd think you could also use `bytesplice`, but it actually has the same issue:

```ruby
Post = Struct.new(:title, :body) do
  def serialize(buf)
    buf << 255 << title.bytesize
    buf.bytesplice(buf.bytesize, title.bytesize, title)
    buf << 255 << body.bytesize
    buf.bytesplice(buf.bytesize, body.bytesize, title)
  end
end
Post.new("H€llo", "Wôrld").serialize("somedata".b) # => 'String#bytesplice': incompatible character encodings: BINARY (ASCII-8BIT) and UTF-8 (Encoding::CompatibilityError)
```

And even if it worked, it would be very unergonomic.

### Proposal: a `byteconcat` method

A solution to this would be to add a new `byteconcat` method, that could be shimed as:

```ruby
class String
  def byteconcat(*strings)
    strings.map! do |s|
      if s.is_a?(String) && s.encoding != encoding
        s.dup.force_encoding(encoding)
      else
        s
      end
    end
    concat(*strings)
  end
end

Post = Struct.new(:title, :body) do
  def serialize(buf)
    buf.byteconcat(
      255, title.bytesize, title,
      255, body.bytesize, body,
    )
  end
end

Post.new("H€llo", "Wôrld").serialize("somedata".b) # => "somedata\xFF\aH\xE2\x82\xACllo\xFF\x06W\xC3\xB4rld" #<Encoding:ASCII-8BIT>
```

But of course a builtin implementation wouldn't need to dup the arguments.

Like other `byte*` methods, it's the responsibility of the caller to ensure the resulting string has a valid encoding, or
to deal with it if not.

### Method name and signature

#### Name

This proposal suggests `String#byteconcat`, to mirror `String#concat`, but other names are possible:

  - `byteappend` (like `Array#append`)
  - `bytepush`  (like `Array#push`)

#### Signature

This proposal makes `byteconcat` accept either `String` or `Integer` (in char range) arguments like `concat`. I believe it makes sense for consistency and also because it's not uncommon for protocols to have some byte based segments, and Integers are more convenient there.

The proposed method also accept variable arguments for consistency with `String#concat`, `Array#push`, `Array#append`.

The proposed method returns self, like `concat` and others.

### YJIT consideration

I consulted @maximecb about this proposal, and according to her, accepting variable arguments makes it harder for YJIT to optimize.
I suspect consistency with other APIs trumps the performance consideration, but I think it's worth mentioning.

-- 
https://bugs.ruby-lang.org/