Issue #19291 has been reported by tompng (tomoya ishida).
----------------------------------------
Bug #19291: Ripper's on_parse_error token has wrong content
https://bugs.ruby-lang.org/issues/19291
* Author: tompng (tomoya ishida)
* Status: Open
* Priority: Normal
* ruby -v: ruby 3.2.0 (2022-12-25 revision a528908271) [x86_64-darwin20]
* Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN
----------------------------------------
Ruby 3.1
~~~ruby
Ripper::Lexer.new("{a:,aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\nhelloworld").scan.map(&:tok)
# =>
["{", "a:", ",",
"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "", "\n",
"helloworld"]
~~~
Ruby 3.2.0
~~~ruby
Ripper::Lexer.new("{a:,aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\nhelloworld").scan.map(&:tok)
# =>
["{",
"a:",
",",
"oworld\u0000\u0000\u0000\u0000\u0000\u0000z`\u0001\u0000\u0000\u0000\u0000\u0000\xC0\xF0$\u0010\u0001\u0000\u0000\u0000\x80A\xB8\u0019\xB5\u007F\u0000\u0000\x91\x94",
"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"\n",
"helloworld"]
~~~
--
https://bugs.ruby-lang.org/