Issue #19736 has been reported by tompng (tomoya ishida). ---------------------------------------- Bug #19736: Ripper.tokenize ignores token after heredoc_beg when embexpr inside heredoc is unclosed https://bugs.ruby-lang.org/issues/19736 * Author: tompng (tomoya ishida) * Status: Open * Priority: Normal * ruby -v: ruby 3.3.0dev (2023-06-19T09:28:10Z master 2a80bac9f0) [x86_64-linux] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- I expect Ripper.tokenize(code) to tokenize `+1+2+3\n` part. ```ruby code = <<'RUBY' <<A+1+2+3 #{4 RUBY Ripper.tokenize code # => ["<<A", "\#{", "4", "\n"] ``` It will be easy for IRB's internal code to handle indent and other things if Ripper.tokenize can tokenize all part of the code even if it is incomplete. -- https://bugs.ruby-lang.org/