[ruby-core:123971] [Ruby Bug#21756] Ripper fails to parse pathological heredoc
Issue #21756 has been reported by Earlopain (Earlopain _). ---------------------------------------- Bug #21756: Ripper fails to parse pathological heredoc https://bugs.ruby-lang.org/issues/21756 * Author: Earlopain (Earlopain _) * Status: Open * ruby -v: ruby 4.0.0dev (2025-11-28T10:49:46Z master dcb9e17f46) +PRISM [x86_64-linux] * Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN ---------------------------------------- ```rb # test.rb pp <<-A, %w[j\ i A j] ``` ```sh $ RBENV_VERSION=ruby-dev ruby -rripper -ve "Ripper.lex(File.read('test.rb'), raise_errors: true)" ruby 4.0.0dev (2025-11-28T10:49:46Z master dcb9e17f46) +PRISM [x86_64-linux] /home/user/.rbenv/versions/ruby-dev/lib/ruby/4.0.0+0/ripper/lexer.rb:180:in 'Ripper::Lexer#parse': syntax error, unexpected literal content, expecting ' ' (SyntaxError) from /home/user/.rbenv/versions/ruby-dev/lib/ruby/4.0.0+0/ripper/lexer.rb:155:in 'Ripper::Lexer#lex' from /home/user/.rbenv/versions/ruby-dev/lib/ruby/4.0.0+0/ripper/lexer.rb:52:in 'Ripper.lex' from -e:1:in '<main>' ``` Both prism and parse.y are able to interpret this code: ```rb $ RBENV_VERSION=ruby-dev ruby test.rb "i\n" ["j\n", "j"] $ RBENV_VERSION=ruby-dev ruby --parser=parse.y test.rb "i\n" ["j\n" + "j"] ``` -- https://bugs.ruby-lang.org/
Issue #21756 has been updated by tompng (tomoya ishida). Looks like there is one more separate issue here. Prism and parse.y execution result differs. ~~~ruby p <<-A, %w[a\ A b c] # Prism: ["a\n", "b", "c"] # parse.y: ["a\nb", "c"] ~~~ Changing `%w` to `%W`, Prism and parse.y are the same ~~~ruby p <<-A, %W[a\ A b c] # => Both ["a\nb", "c"] ~~~ ---------------------------------------- Bug #21756: Ripper fails to parse pathological heredoc https://bugs.ruby-lang.org/issues/21756#change-115413 * Author: Earlopain (Earlopain _) * Status: Open * ruby -v: ruby 4.0.0dev (2025-11-28T10:49:46Z master dcb9e17f46) +PRISM [x86_64-linux] * Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN ---------------------------------------- ```rb # test.rb pp <<-A, %w[j\ i A j] ``` ```sh $ RBENV_VERSION=ruby-dev ruby -rripper -ve "Ripper.lex(File.read('test.rb'), raise_errors: true)" ruby 4.0.0dev (2025-11-28T10:49:46Z master dcb9e17f46) +PRISM [x86_64-linux] /home/user/.rbenv/versions/ruby-dev/lib/ruby/4.0.0+0/ripper/lexer.rb:180:in 'Ripper::Lexer#parse': syntax error, unexpected literal content, expecting ' ' (SyntaxError) from /home/user/.rbenv/versions/ruby-dev/lib/ruby/4.0.0+0/ripper/lexer.rb:155:in 'Ripper::Lexer#lex' from /home/user/.rbenv/versions/ruby-dev/lib/ruby/4.0.0+0/ripper/lexer.rb:52:in 'Ripper.lex' from -e:1:in '<main>' ``` Both prism and parse.y are able to interpret this code: ```rb $ RBENV_VERSION=ruby-dev ruby test.rb "i\n" ["j\n", "j"] $ RBENV_VERSION=ruby-dev ruby --parser=parse.y test.rb "i\n" ["j\n" + "j"] ``` -- https://bugs.ruby-lang.org/
Issue #21756 has been updated by Earlopain (Earlopain _). Ah, yeah. It's not expecting disjointed strings in arrays where interpolation is not allowed. That doesn't hold with line continuations and heredocs. I'm working on a fix, should not be so difficult. ---------------------------------------- Bug #21756: Ripper fails to parse pathological heredoc https://bugs.ruby-lang.org/issues/21756#change-115415 * Author: Earlopain (Earlopain _) * Status: Open * ruby -v: ruby 4.0.0dev (2025-11-28T10:49:46Z master dcb9e17f46) +PRISM [x86_64-linux] * Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN ---------------------------------------- ```rb # test.rb pp <<-A, %w[j\ i A j] ``` ```sh $ RBENV_VERSION=ruby-dev ruby -rripper -ve "Ripper.lex(File.read('test.rb'), raise_errors: true)" ruby 4.0.0dev (2025-11-28T10:49:46Z master dcb9e17f46) +PRISM [x86_64-linux] /home/user/.rbenv/versions/ruby-dev/lib/ruby/4.0.0+0/ripper/lexer.rb:180:in 'Ripper::Lexer#parse': syntax error, unexpected literal content, expecting ' ' (SyntaxError) from /home/user/.rbenv/versions/ruby-dev/lib/ruby/4.0.0+0/ripper/lexer.rb:155:in 'Ripper::Lexer#lex' from /home/user/.rbenv/versions/ruby-dev/lib/ruby/4.0.0+0/ripper/lexer.rb:52:in 'Ripper.lex' from -e:1:in '<main>' ``` Both prism and parse.y are able to interpret this code: ```rb $ RBENV_VERSION=ruby-dev ruby test.rb "i\n" ["j\n", "j"] $ RBENV_VERSION=ruby-dev ruby --parser=parse.y test.rb "i\n" ["j\n" + "j"] ``` -- https://bugs.ruby-lang.org/
Issue #21756 has been updated by Earlopain (Earlopain _). For the prism part of the issue https://github.com/ruby/prism/pull/3778 ---------------------------------------- Bug #21756: Ripper fails to parse pathological heredoc https://bugs.ruby-lang.org/issues/21756#change-115422 * Author: Earlopain (Earlopain _) * Status: Open * ruby -v: ruby 4.0.0dev (2025-11-28T10:49:46Z master dcb9e17f46) +PRISM [x86_64-linux] * Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN ---------------------------------------- ```rb # test.rb pp <<-A, %w[j\ i A j] ``` ```sh $ RBENV_VERSION=ruby-dev ruby -rripper -ve "Ripper.lex(File.read('test.rb'), raise_errors: true)" ruby 4.0.0dev (2025-11-28T10:49:46Z master dcb9e17f46) +PRISM [x86_64-linux] /home/user/.rbenv/versions/ruby-dev/lib/ruby/4.0.0+0/ripper/lexer.rb:180:in 'Ripper::Lexer#parse': syntax error, unexpected literal content, expecting ' ' (SyntaxError) from /home/user/.rbenv/versions/ruby-dev/lib/ruby/4.0.0+0/ripper/lexer.rb:155:in 'Ripper::Lexer#lex' from /home/user/.rbenv/versions/ruby-dev/lib/ruby/4.0.0+0/ripper/lexer.rb:52:in 'Ripper.lex' from -e:1:in '<main>' ``` Both prism and parse.y are able to interpret this code: ```rb $ RBENV_VERSION=ruby-dev ruby test.rb "i\n" ["j\n", "j"] $ RBENV_VERSION=ruby-dev ruby --parser=parse.y test.rb "i\n" ["j\n" + "j"] ``` -- https://bugs.ruby-lang.org/
participants (2)
-
Earlopain (Earlopain _) -
tompng (tomoya ishida)