Issue #21528 has been updated by byroot (Jean Boussier).

I'm not sure how to correctly fix it due to lack of knowledge about safe byte truncation.

If that helps, I did something similar in ruby/json: https://github.com/ruby/json/blob/3090a63a956c30e6d30d93fc9667deccd5e31327/e... / https://github.com/ruby/json/commit/e144793b7226c2df75c414749d6f87ab7fcf4dce It's not perfect as it doesn't consider grapheme clusters, but at least it ensures the included snippet is valid UTF-8. ---------------------------------------- Bug #21528: SyntaxError#message may have broken encoding with multibyte source under Prism https://bugs.ruby-lang.org/issues/21528#change-114211 * Author: alpaca-tc (Hiroyuki Ishii) * Status: Open * Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN ---------------------------------------- Since the introduction of Prism, when parsing Ruby source code that contains multibyte characters, SyntaxError#message can sometimes have invalid encoding. Here is a reproducible example: ``` begin RubyVM::InstructionSequence.compile(<<~CODE, nil, nil, 1) if a # 0000000000000ああああああ # CODE rescue SyntaxError => e $e = e puts e.message # string contains a multibyte character that is cut off mid-byte. \xE3 # <compiled>:3: syntax errors found # 1 | if a # > 2 | # 0000000000000あああああ\xE3 ... # | ^ expected an `end` to close the conditional clause # > 3 | # # | ^ unexpected end-of-input, assuming it is closing the parent top level context puts e.message.valid_encoding? #=> expected true, but got false end ``` This appears to be caused by a truncation process in prism's error message generating that does not consider multibyte character boundaries. See: The truncation logic around [prism_compile.c L10696-L10709](https://github.com/ruby/ruby//blob/30a20bc166bc37acd7dcb3788686df149c7f428a/...) I'm not sure how to correctly fix it due to lack of knowledge about safe byte truncation. I discovered this issue through irb, which attempts to display source code even when it contains syntax errors. Because irb uses `SyntaxError#message`, it raised an `ArgumentError: invalid byte sequence in UTF-8`. See: https://github.com/ruby/irb/blob/f60dfa8549f746f69e9a6d160604a7a4974ffac1/li... If this is considered an irb issue, I already have a patch for IRB that handles it. -- https://bugs.ruby-lang.org/

Issue #21528 has been updated by ko1 (Koichi Sasada). Assignee set to prism ---------------------------------------- Bug #21528: SyntaxError#message may have broken encoding with multibyte source under Prism https://bugs.ruby-lang.org/issues/21528#change-114302 * Author: alpaca-tc (Hiroyuki Ishii) * Status: Open * Assignee: prism * Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN ---------------------------------------- Since the introduction of Prism, when parsing Ruby source code that contains multibyte characters, SyntaxError#message can sometimes have invalid encoding. Here is a reproducible example: ``` begin RubyVM::InstructionSequence.compile(<<~CODE, nil, nil, 1) if a # 0000000000000ああああああ # CODE rescue SyntaxError => e $e = e puts e.message # string contains a multibyte character that is cut off mid-byte. \xE3 # <compiled>:3: syntax errors found # 1 | if a # > 2 | # 0000000000000あああああ\xE3 ... # | ^ expected an `end` to close the conditional clause # > 3 | # # | ^ unexpected end-of-input, assuming it is closing the parent top level context puts e.message.valid_encoding? #=> expected true, but got false end ``` This appears to be caused by a truncation process in prism's error message generating that does not consider multibyte character boundaries. See: The truncation logic around [prism_compile.c L10696-L10709](https://github.com/ruby/ruby//blob/30a20bc166bc37acd7dcb3788686df149c7f428a/...) I'm not sure how to correctly fix it due to lack of knowledge about safe byte truncation. I discovered this issue through irb, which attempts to display source code even when it contains syntax errors. Because irb uses `SyntaxError#message`, it raised an `ArgumentError: invalid byte sequence in UTF-8`. See: https://github.com/ruby/irb/blob/f60dfa8549f746f69e9a6d160604a7a4974ffac1/li... If this is considered an irb issue, I already have a patch for IRB that handles it. -- https://bugs.ruby-lang.org/