[ruby-core:118132] [Ruby master Bug#20517] `Ripper.tokenize('"\\M-あ"')` separates encoding valid string to encoding invalid string.
Issue #20517 has been reported by tompng (tomoya ishida). ---------------------------------------- Bug #20517: `Ripper.tokenize('"\\M-あ"')` separates encoding valid string to encoding invalid string. https://bugs.ruby-lang.org/issues/20517 * Author: tompng (tomoya ishida) * Status: Open * ruby -v: ruby 3.4.0dev (2024-04-11T08:57:52Z master e7f8db9079) [x86_64-linux] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- ~~~ruby Ripper.tokenize '"\\M-あ"' => ["\"", "\\M-\xE3", "\x81", "\x82", "\""] ~~~ I expect all tokens to be valid_encoding if the source string is valid_encoding. Similar to https://bugs.ruby-lang.org/issues/20030 -- https://bugs.ruby-lang.org/
Issue #20517 has been updated by nobu (Nobuyoshi Nakada). https://github.com/ruby/ruby/pull/10893 ---------------------------------------- Bug #20517: `Ripper.tokenize('"\\M-あ"')` separates encoding valid string to encoding invalid string. https://bugs.ruby-lang.org/issues/20517#change-108573 * Author: tompng (tomoya ishida) * Status: Open * ruby -v: ruby 3.4.0dev (2024-04-11T08:57:52Z master e7f8db9079) [x86_64-linux] * Backport: 3.1: REQUIRED, 3.2: REQUIRED, 3.3: REQUIRED ---------------------------------------- ~~~ruby Ripper.tokenize '"\\M-あ"' => ["\"", "\\M-\xE3", "\x81", "\x82", "\""] ~~~ I expect all tokens to be valid_encoding if the source string is valid_encoding. Similar to https://bugs.ruby-lang.org/issues/20030 -- https://bugs.ruby-lang.org/
Issue #20517 has been updated by k0kubun (Takashi Kokubun). Backport changed from 3.1: REQUIRED, 3.2: REQUIRED, 3.3: REQUIRED to 3.1: REQUIRED, 3.2: REQUIRED, 3.3: DONE ruby_3_3 commit:05553cf22d43dd78b8f30cc4591230b5c000c538. ---------------------------------------- Bug #20517: `Ripper.tokenize('"\\M-あ"')` separates encoding valid string to encoding invalid string. https://bugs.ruby-lang.org/issues/20517#change-108622 * Author: tompng (tomoya ishida) * Status: Closed * ruby -v: ruby 3.4.0dev (2024-04-11T08:57:52Z master e7f8db9079) [x86_64-linux] * Backport: 3.1: REQUIRED, 3.2: REQUIRED, 3.3: DONE ---------------------------------------- ~~~ruby Ripper.tokenize '"\\M-あ"' => ["\"", "\\M-\xE3", "\x81", "\x82", "\""] ~~~ I expect all tokens to be valid_encoding if the source string is valid_encoding. Similar to https://bugs.ruby-lang.org/issues/20030 -- https://bugs.ruby-lang.org/
Issue #20517 has been updated by nagachika (Tomoyuki Chikanaga). Backport changed from 3.1: REQUIRED, 3.2: REQUIRED, 3.3: DONE to 3.1: REQUIRED, 3.2: DONE, 3.3: DONE ruby_3_2 commit:96a82418b2efe98a92f239a9a1cbf30dd396d335 merged revision(s) commit:05553cf22d43dd78b8f30cc4591230b5c000c538. ---------------------------------------- Bug #20517: `Ripper.tokenize('"\\M-あ"')` separates encoding valid string to encoding invalid string. https://bugs.ruby-lang.org/issues/20517#change-109137 * Author: tompng (tomoya ishida) * Status: Closed * ruby -v: ruby 3.4.0dev (2024-04-11T08:57:52Z master e7f8db9079) [x86_64-linux] * Backport: 3.1: REQUIRED, 3.2: DONE, 3.3: DONE ---------------------------------------- ~~~ruby Ripper.tokenize '"\\M-あ"' => ["\"", "\\M-\xE3", "\x81", "\x82", "\""] ~~~ I expect all tokens to be valid_encoding if the source string is valid_encoding. Similar to https://bugs.ruby-lang.org/issues/20030 -- https://bugs.ruby-lang.org/
participants (4)
-
k0kubun (Takashi Kokubun) -
nagachika (Tomoyuki Chikanaga)
-
nobu (Nobuyoshi Nakada)
-
tompng (tomoya ishida)