Issue #20189 has been updated by tompng (tomoya ishida). I think so. sjis char does not end with null bytes, other encoding seems same too. ~~~ruby Encoding.list.select {|e| 256.times.any? do |first_byte| a = first_byte.chr b = a + "\0"; # only one of \x??\x00 and \x?? is valid a.force_encoding(e).valid_encoding? != b.force_encoding(e).valid_encoding? end } # => [#<Encoding:UTF-16BE>, #<Encoding:UTF-16LE>] ~~~ It looks like there is no string like ("表"(sjis)=="\x95\x5c") that satisfies "\x??\x00" is valid and "\x??" is not. I opened a pull request https://github.com/ruby/ruby/pull/9552 ---------------------------------------- Bug #20189: `rb_str_resize` does not clear coderange when expanding https://bugs.ruby-lang.org/issues/20189#change-106247 * Author: tompng (tomoya ishida) * Status: Open * Priority: Normal * ruby -v: ruby 3.4.0dev (2024-01-09T07:07:19Z master db476cc71c) [x86_64-linux] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- Expanding string in some encoding (utf16 utf32) can change coderange to either valid or broken, but rb_str_resize does not clear coderange. This will cause a bug in c-extension libraries that use rb_str_resize. ~~~ruby # Example for stringio s = StringIO.new("\0".encode('UTF-16LE')) s.truncate(1); s.truncate(2); s.string.valid_encoding? #=> true s.truncate(1); s.string.valid_encoding?; s.truncate(2); s.string.valid_encoding? #=> false (expect to be true) ~~~ -- https://bugs.ruby-lang.org/

Issue #20189 has been updated by byroot (Jean Boussier).

Expanding string in some encoding (utf16 utf32) can change coderange to either valid or broken,

I must admit I'm not very familiar with wide char encodings, but this surprises me a bit. Ruby strings should always have their terminator, so I don't see how expanding a string would change their interpretation. ---------------------------------------- Bug #20189: `rb_str_resize` does not clear coderange when expanding https://bugs.ruby-lang.org/issues/20189#change-106250 * Author: tompng (tomoya ishida) * Status: Open * Priority: Normal * ruby -v: ruby 3.4.0dev (2024-01-09T07:07:19Z master db476cc71c) [x86_64-linux] * Backport: 3.0: DONTNEED, 3.1: DONTNEED, 3.2: REQUIRED, 3.3: REQUIRED ---------------------------------------- Expanding string in some encoding (utf16 utf32) can change coderange to either valid or broken, but rb_str_resize does not clear coderange. This will cause a bug in c-extension libraries that use rb_str_resize. ~~~ruby # Example for stringio s = StringIO.new("\0".encode('UTF-16LE')) s.truncate(1); s.truncate(2); s.string.valid_encoding? #=> true s.truncate(1); s.string.valid_encoding?; s.truncate(2); s.string.valid_encoding? #=> false (expect to be true) ~~~ -- https://bugs.ruby-lang.org/

Issue #20189 has been updated by wanabe (_ wanabe). Status changed from Open to Closed https://github.com/ruby/ruby/pull/9552 has already been merged in commit 9c7374b0e6cbfb00e4f71d0b2eddba128ba452f7. It seems it was left open for some reason, but I'm closing it now. ---------------------------------------- Bug #20189: `rb_str_resize` does not clear coderange when expanding https://bugs.ruby-lang.org/issues/20189#change-118538 * Author: tompng (tomoya ishida) * Status: Closed * ruby -v: ruby 3.4.0dev (2024-01-09T07:07:19Z master db476cc71c) [x86_64-linux] * Backport: 3.0: DONTNEED, 3.1: DONTNEED, 3.2: REQUIRED, 3.3: REQUIRED ---------------------------------------- Expanding string in some encoding (utf16 utf32) can change coderange to either valid or broken, but rb_str_resize does not clear coderange. This will cause a bug in c-extension libraries that use rb_str_resize. ~~~ruby # Example for stringio s = StringIO.new("\0".encode('UTF-16LE')) s.truncate(1); s.truncate(2); s.string.valid_encoding? #=> true s.truncate(1); s.string.valid_encoding?; s.truncate(2); s.string.valid_encoding? #=> false (expect to be true) ~~~ -- https://bugs.ruby-lang.org/