Issue #20030 has been updated by nobu (Nobuyoshi Nakada).
https://github.com/ruby/ruby/pull/9091
This fixes another ripper scanner event issue at a syntax error.
----------------------------------------
Bug #20030: `Ripper.tokenize('"\\C-あ"')` separates encoding valid string
to encoding invalid string.
https://bugs.ruby-lang.org/issues/20030#change-105495
* Author: tompng (tomoya ishida)
* Status: Open
* Priority: Normal
* ruby -v: ruby 3.3.0dev (2023-11-30T16:23:25Z master d048bae96b) [x86_64-linux]
* Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN
----------------------------------------
~~~ruby
Ripper.tokenize '"\\C-あ"' # or Ripper.tokenize
"\"\\C-\u3042\""
# =>
["\"", "\x81", "\x82", "\""]
~~~
I expect all tokens to be valid_encoding if the source string is valid_encoding.
This is causing IRB crash when typing `"\C-あ"`.
--
https://bugs.ruby-lang.org/