Issue #20030 has been updated by nobu (Nobuyoshi Nakada). https://github.com/ruby/ruby/pull/9091 This fixes another ripper scanner event issue at a syntax error. ---------------------------------------- Bug #20030: `Ripper.tokenize('"\\C-あ"')` separates encoding valid string to encoding invalid string. https://bugs.ruby-lang.org/issues/20030#change-105495 * Author: tompng (tomoya ishida) * Status: Open * Priority: Normal * ruby -v: ruby 3.3.0dev (2023-11-30T16:23:25Z master d048bae96b) [x86_64-linux] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- ~~~ruby Ripper.tokenize '"\\C-あ"' # or Ripper.tokenize "\"\\C-\u3042\"" # => ["\"", "\x81", "\x82", "\""] ~~~ I expect all tokens to be valid_encoding if the source string is valid_encoding. This is causing IRB crash when typing `"\C-あ"`. -- https://bugs.ruby-lang.org/