Issue #20434 has been updated by shyouhei (Shyouhei Urabe). Subject changed from Deprecate regular expression modifiers to Deprecate encoding-releated regular expression modifiers +1 for deprecating encoding modifiers, but they're not everything that a regexp can take. For instance `/foo/i` is a valid regular expression literal in ruby, perl, PHP(preg), and Javascript. I'm sure Kevin didn't intend to kill everything. Let me narrow the scope of this request; subject updated. ---------------------------------------- Misc #20434: Deprecate encoding-releated regular expression modifiers https://bugs.ruby-lang.org/issues/20434#change-107981 * Author: kddnewton (Kevin Newton) * Status: Open ---------------------------------------- This is a follow-up to @duerst's comment here: https://bugs.ruby-lang.org/issues/20406#note-6. As noted in the other issue, there are many encodings that factor in to how a regular expression operates. This includes: * The encoding of the file * The encoding of the string parts within the regular expression * The regular expression encoding modifiers * The encoding of the string being matched At the time the modifiers were introduced, I believe the modifiers may have been the only (??) encoding that factored in here. At this point, however, they can lead to quite a bit of confusion, as noted in the other ticket. I would like to propose to deprecate the regular expression encoding modifiers. Instead, we could suggest in a warning to instead create a regular expression with an encoded string. For example, when we find: ```ruby /\x81\x40/s ``` we would instead suggest: ```ruby ::Regexp.new(::String.new("\x81\x40", encoding: "Windows-31J")) ``` or equivalent. As a migration path, we could do the following: 1. Emit a warning to change to the suggested expression 2. Change the compiler to compile to the suggested expression when those flags are found 3. Remove support for the flags Step 2 may be unnecessary depending on how long of a timeline we would like to provide. To be clear, I'm not advocating for any particular timeline, and would be fine with this being multiple years/versions to give plenty of time for people to migrate. But I do think this would be a good change to eliminate confusion about the interaction between the four different encodings at play. -- https://bugs.ruby-lang.org/

Issue #20434 has been updated by byroot (Jean Boussier). `/\x81\x40/.force_encoding("Windows-31J")` wouldn't work because `String#force_encoding` mutates the string, and Regexp literals are immutable. Similarly `String#encode` doesn't just change the string encoding attribute, but convert the bytes to the new encoding. So I'd expect `/\3000/.encode("Windows-31J")` to fail with: ```ruby \x81" on UTF-8 (Encoding::InvalidByteSequenceError) ``` So I think the String API to mirror would be `String.new(encoding:)` - `Regexp.new(/\x81\x40/, encoding: Encoding::WINDOWS_31J)` - `Regexp.new("\x81\x40", encoding: Encoding::WINDOWS_31J)` But if we want an instance method, I think something like: `/\x81\x40/.encoded(Encoding::WINDOWS_31J)`, which by the way would also be useful on `String`, e.g., this is common: ```ruby # frozen_string_literal: true THING = "fée".dup.force_encoding(Encoding::ISO8859_1) ``` So it could become: ```ruby # frozen_string_literal: true THING = "fée".encoded(Encoding::ISO8859_1) ``` ---------------------------------------- Misc #20434: Deprecate encoding-releated regular expression modifiers https://bugs.ruby-lang.org/issues/20434#change-107997 * Author: kddnewton (Kevin Newton) * Status: Open ---------------------------------------- This is a follow-up to @duerst's comment here: https://bugs.ruby-lang.org/issues/20406#note-6. As noted in the other issue, there are many encodings that factor in to how a regular expression operates. This includes: * The encoding of the file * The encoding of the string parts within the regular expression * The regular expression encoding modifiers * The encoding of the string being matched At the time the modifiers were introduced, I believe the modifiers may have been the only (??) encoding that factored in here. At this point, however, they can lead to quite a bit of confusion, as noted in the other ticket. I would like to propose to deprecate the regular expression encoding modifiers. Instead, we could suggest in a warning to instead create a regular expression with an encoded string. For example, when we find: ```ruby /\x81\x40/s ``` we would instead suggest: ```ruby ::Regexp.new(::String.new("\x81\x40", encoding: "Windows-31J")) ``` or equivalent. As a migration path, we could do the following: 1. Emit a warning to change to the suggested expression 2. Change the compiler to compile to the suggested expression when those flags are found 3. Remove support for the flags Step 2 may be unnecessary depending on how long of a timeline we would like to provide. To be clear, I'm not advocating for any particular timeline, and would be fine with this being multiple years/versions to give plenty of time for people to migrate. But I do think this would be a good change to eliminate confusion about the interaction between the four different encodings at play. -- https://bugs.ruby-lang.org/

Issue #20434 has been updated by kddnewton (Kevin Newton). Thanks @shyouhei — I definitely only meant encoding-relate modifiers. I really like the other ones! ---------------------------------------- Misc #20434: Deprecate encoding-related regular expression modifiers https://bugs.ruby-lang.org/issues/20434#change-108009 * Author: kddnewton (Kevin Newton) * Status: Open ---------------------------------------- This is a follow-up to @duerst's comment here: https://bugs.ruby-lang.org/issues/20406#note-6. As noted in the other issue, there are many encodings that factor in to how a regular expression operates. This includes: * The encoding of the file * The encoding of the string parts within the regular expression * The regular expression encoding modifiers * The encoding of the string being matched At the time the modifiers were introduced, I believe the modifiers may have been the only (??) encoding that factored in here. At this point, however, they can lead to quite a bit of confusion, as noted in the other ticket. I would like to propose to deprecate the regular expression encoding modifiers. Instead, we could suggest in a warning to instead create a regular expression with an encoded string. For example, when we find: ```ruby /\x81\x40/s ``` we would instead suggest: ```ruby ::Regexp.new(::String.new("\x81\x40", encoding: "Windows-31J")) ``` or equivalent. As a migration path, we could do the following: 1. Emit a warning to change to the suggested expression 2. Change the compiler to compile to the suggested expression when those flags are found 3. Remove support for the flags Step 2 may be unnecessary depending on how long of a timeline we would like to provide. To be clear, I'm not advocating for any particular timeline, and would be fine with this being multiple years/versions to give plenty of time for people to migrate. But I do think this would be a good change to eliminate confusion about the interaction between the four different encodings at play. -- https://bugs.ruby-lang.org/