Issue #20579 has been reported by tompng (tomoya ishida). ---------------------------------------- Bug #20579: `Ripper.tokenize("1\n ")` does not include trailing spaces https://bugs.ruby-lang.org/issues/20579 * Author: tompng (tomoya ishida) * Status: Open * ruby -v: ruby 3.4.0dev (2024-06-13T09:49:46Z master 8b843b0dc7) [x86_64-linux] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- ~~~ruby # trailing :on_sp, :on_ignored_nl are included in the result Ripper.tokenize("1\n\t \t\n\t \t") # => ["1", "\n", "\t \t", "\n", "\t \t"] # trailing :on_sp disappears Ripper.tokenize("1\n\t \t") # => ["1", "\n"] ~~~ Found with `Ripper.tokenize(random_string).join != random_string` while investigating https://bugs.ruby-lang.org/issues/20578 -- https://bugs.ruby-lang.org/