Issue #21870 has been updated by tompng (tomoya ishida). I found 130 (5 sets of 26 alphabets) characters matching both `\p{S}` and `\p{Word}`. The visual looks like alphabet-ish symbol character ~~~ruby (0..0x10ffff).select{(s=''<<it; s=~/\p{Word}/&&s=~/\p{S}/) rescue false}.map{''<<it}.join # ⒶⒷⒸⒹⒺⒻⒼⒽⒾⒿⓀⓁⓂⓃⓄⓅⓆⓇⓈⓉⓊⓋⓌⓍⓎⓏ # ⓐⓑⓒⓓⓔⓕⓖⓗⓘⓙⓚⓛⓜⓝⓞⓟⓠⓡⓢⓣⓤⓥⓦⓧⓨⓩ # 🄰🄱🄲🄳🄴🄵🄶🄷🄸🄹🄺🄻🄼🄽🄾🄿🅀🅁🅂🅃🅄🅅🅆🅇🅈🅉 # 🅐🅑🅒🅓🅔🅕🅖🅗🅘🅙🅚🅛🅜🅝🅞🅟🅠🅡🅢🅣🅤🅥🅦🅧🅨🅩 # 🅰🅱🅲🅳🅴🅵🅶🅷🅸🅹🅺🅻🅼🅽🅾🅿🆀🆁🆂🆃🆄🆅🆆🆇🆈🆉 ~~~ I'm not sure how to read unicode properties, but it looks like these characters are Alphabetic:Yes and also in Other_Symbol category https://util.unicode.org/UnicodeJsps/character.jsp?a=%E2%92%B6 ---------------------------------------- Bug #21870: Regexp: Warnings when using multiple non-overlapping \p{...} classes https://bugs.ruby-lang.org/issues/21870#change-116315 * Author: jneen (Jeanine Adkisson) * Status: Open * ruby -v: 4.0.1 * Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN, 4.0: UNKNOWN ---------------------------------------- ```ruby $VERBOSE = true # warning: character class has duplicated range: /[\p{Word}\p{S}]/ regex = /[\p{Word}\p{S}]/ ``` As far as I can tell this is a perfectly valid and non-overlapping set of unicode properties, but I am still being spammed with warnings. Using `/(\p{Word}|\p{S})/` is kind of a workaround, but it is slower. -- https://bugs.ruby-lang.org/

Issue #21870 has been updated by jneen (Jeanine Adkisson). Another example of this is `/[\p{Word}\p{Cf}]/`, which seem to overlap precisely on ZWNJ (U+200C) and ZWJ (U+200D). ```ruby [1] pry(main)> (0..0x10ffff).select{(s=[it].pack('U'); s=~/\p{Word}/&&s=~/\p{Cf}/) rescue false}.map{it.to_s 16 } => ["200c", "200d"] [2] pry(main)> /[\p{Word}\p{Cf}]/ (pry):5: warning: character class has duplicated range: /[\p{Word}\p{Cf}]/ => /[\p{Word}\p{Cf}]/ [3] pry(main)> ``` ---------------------------------------- Bug #21870: Regexp: Warnings when using slightly overlapping \p{...} classes https://bugs.ruby-lang.org/issues/21870#change-116324 * Author: jneen (Jeanine Adkisson) * Status: Open * ruby -v: 4.0.1 * Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN, 4.0: UNKNOWN ---------------------------------------- ```ruby $VERBOSE = true # warning: character class has duplicated range: /[\p{Word}\p{S}]/ regex = /[\p{Word}\p{S}]/ ``` As far as I can tell this is a perfectly valid ~~and non-overlapping~~ set of unicode properties, but I am still being spammed with warnings. Using `/(\p{Word}|\p{S})/` is kind of a workaround, but it is slower. Edit: They do overlap somewhat, but I think the deeper issue is there is not a convenient way to express this without falling back to raw unicode ranges. -- https://bugs.ruby-lang.org/

Issue #21870 has been updated by mame (Yusuke Endoh). jneen (Jeanine Adkisson) wrote in #note-7:

That specific case also appears to have changed, e.g. on 3.4.1:

It is an intentional bug fix. See #21503. While I understand your trouble, this warning is functioning exactly as intended. How do you suggest resolving it? ---------------------------------------- Bug #21870: Regexp: Warnings when using slightly overlapping \p{...} classes https://bugs.ruby-lang.org/issues/21870#change-116328 * Author: jneen (Jeanine Adkisson) * Status: Open * ruby -v: 4.0.1 * Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN, 4.0: UNKNOWN ---------------------------------------- ```ruby $VERBOSE = true # warning: character class has duplicated range: /[\p{Word}\p{S}]/ regex = /[\p{Word}\p{S}]/ ``` As far as I can tell this is a perfectly valid ~~and non-overlapping~~ set of unicode properties, but I am still being spammed with warnings. Using `/(\p{Word}|\p{S})/` is kind of a workaround, but it is slower. Edit: They do overlap somewhat, but I think the deeper issue is there is not a convenient way to express this without falling back to raw unicode ranges. -- https://bugs.ruby-lang.org/

Issue #21870 has been updated by kddnewton (Kevin Newton). This might be a good opportunity to add the `||` operator from the Unicode spec (https://www.unicode.org/reports/tr18/#Subtraction_and_Intersection. We could make that one not warn, because it's explicitly desired. As in: ```ruby $VERBOSE = true regex = /[\p{Word}\p{S}]/ # warning regex = /[\p{Word}||\p{S}]/ # no warning ``` ---------------------------------------- Bug #21870: Regexp: Warnings when using slightly overlapping \p{...} classes https://bugs.ruby-lang.org/issues/21870#change-116338 * Author: jneen (Jeanine Adkisson) * Status: Open * ruby -v: 4.0.1 * Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN, 4.0: UNKNOWN ---------------------------------------- ```ruby $VERBOSE = true # warning: character class has duplicated range: /[\p{Word}\p{S}]/ regex = /[\p{Word}\p{S}]/ ``` As far as I can tell this is a perfectly valid ~~and non-overlapping~~ set of unicode properties, but I am still being spammed with warnings. Using `/(\p{Word}|\p{S})/` is kind of a workaround, but it is slower. Edit: They do overlap somewhat, but I think the deeper issue is there is not a convenient way to express this without falling back to raw unicode ranges. -- https://bugs.ruby-lang.org/

Issue #21870 has been updated by maxfelsher (Max Felsher). If I'm reading the history right, the warning was added in #1831 in order to catch mistakes like a regexp defined as `/[:lower:]/` (as opposed to `/[[:lower:]]/`, I assume). I can see the value in that, but it does seem like there should be a way to list overlapping character classes without a warning (and without turning warnings off completely). ---------------------------------------- Bug #21870: Regexp: Warnings when using slightly overlapping \p{...} classes https://bugs.ruby-lang.org/issues/21870#change-116352 * Author: jneen (Jeanine Adkisson) * Status: Open * ruby -v: 4.0.1 * Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN, 4.0: UNKNOWN ---------------------------------------- ```ruby $VERBOSE = true # warning: character class has duplicated range: /[\p{Word}\p{S}]/ regex = /[\p{Word}\p{S}]/ ``` As far as I can tell this is a perfectly valid ~~and non-overlapping~~ set of unicode properties, but I am still being spammed with warnings. Using `/(\p{Word}|\p{S})/` is kind of a workaround, but it is slower. Edit: They do overlap somewhat, but I think the deeper issue is there is not a convenient way to express this without falling back to raw unicode ranges. -- https://bugs.ruby-lang.org/

Issue #21870 has been updated by jneen (Jeanine Adkisson). Some benchmarks: ```console $ ruby --version ruby 4.0.1 (2026-01-13 revision e04267a14b) +PRISM [arm64-darwin25] ``` ```ruby require 'benchmark' LENGTH = 1000000 REPEAT = 100 TEST_STR = 'a' * LENGTH Benchmark.bm do |bm| bm.report "char class:" do REPEAT.times { /[\p{Word}\p{S}]*/o.match?(TEST_STR) } end bm.report "alternation:" do REPEAT.times { /(?:\p{Word}|\p{S})*/o.match?(TEST_STR) } end end ``` output: ``` user system total real char class: 0.634908 0.302112 0.937020 ( 0.937089) alternation: 0.983069 0.449849 1.432918 ( 1.433005) ``` The alternation syntax is understandably a bit slower, as it would be two nodes in the state machine rather than one unified range test. I expect this effect would be worse when more unicode properties are piled on (as they tend to be in practice), resulting in extra nodes. Either way, `/[\p{Word}\p{S}]/` is a perfectly valid regular expression that as far as I know doesn't have any practical issues, so I don't think it is helpful to warn. ---------------------------------------- Bug #21870: Regexp: Warnings when using slightly overlapping \p{...} classes https://bugs.ruby-lang.org/issues/21870#change-116371 * Author: jneen (Jeanine Adkisson) * Status: Open * ruby -v: 4.0.1 * Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN, 4.0: UNKNOWN ---------------------------------------- ```ruby $VERBOSE = true # warning: character class has duplicated range: /[\p{Word}\p{S}]/ regex = /[\p{Word}\p{S}]/ ``` As far as I can tell this is a perfectly valid ~~and non-overlapping~~ set of unicode properties, but I am still being spammed with warnings. Using `/(\p{Word}|\p{S})/` is kind of a workaround, but it is slower. Edit: They do overlap somewhat, but I think the deeper issue is there is not a convenient way to express this without falling back to raw unicode ranges. -- https://bugs.ruby-lang.org/

Issue #21870 has been updated by jneen (Jeanine Adkisson). Having looked through the onigmo code a bit now, I can think of a few ways forward. **a) Simply don't warn on overlapping ctype classes.** I believe this would only involve removing the check on line 1860 from regparse.c. This would preserve a warning for `/[:foo:]/`, as in #1831, as well as maybe rarer situations like `/[a-fb-g]/`. It would *not* warn on cases like `/[a-z\p{Word}]/` or `/[\p{Alnum}\p{Word}]/`. Whether this is a common enough mistake to warrant a warning I'm not entirely sure. I will also check the performance characteristics of these, in case overlapping ranges is a performance issue (which I doubt, but I think it is best to check). **b) Find a way to check if a character class or range completely subsumes another.** I honestly am not sure how I would go about implementing this, as it is a much deeper check which would require a greater understanding of onigmo internals than I have so far. The idea would be to warn on `/[a-z\p{Word}]/` but *not* on e.g. `/[_-z\p{Word}]`, since the range `_-z` contains a character not matched by `\p{Word}`. This would also catch `/[\p{Alnum}\p{Word}]/`. **c) Rethink the overlapping character warning entirely, and (maybe) more specifically target things like `/[:x:]/`.** This would involve warning only if the first and last character of a char class are literal `:`. Similar to (a), it may turn out that repeated characters in classes are not a performance or correctness issue it is worth warning about at all. But this is a judgment I leave to the team. ---------------------------------------- Bug #21870: Regexp: Warnings when using slightly overlapping \p{...} classes https://bugs.ruby-lang.org/issues/21870#change-116534 * Author: jneen (Jeanine Adkisson) * Status: Open * ruby -v: 4.0.1 * Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN, 4.0: UNKNOWN ---------------------------------------- ```ruby $VERBOSE = true # warning: character class has duplicated range: /[\p{Word}\p{S}]/ regex = /[\p{Word}\p{S}]/ ``` As far as I can tell this is a perfectly valid and non-redundant set of unicode properties, but I am still being spammed with warnings. Using `/(?:\p{Word}|\p{S})/` is kind of a workaround, but it is slower (see benchmarks below), and also less clear. They do overlap somewhat, but I think the deeper issue is there is not a convenient way to express this without falling back to raw unicode ranges. For a similar example, consider `/[\p{Word}\p{Cf}]/`, which overlap precisely on ZWJ and ZWNJ. Even with this very small overlap, Ruby issues a warning, despite neither class being removable without changing the meaning of the regexp. The regexp is valid and as far as I can tell has no practical issues - Onigmo seems to be capable of intersecting overlapping codepoint ranges. This warning was introduced back in 2009 with #1831, to help surface instances of things like `/[:lower:]/` instead of `/[[:lower:]]/`, but even then the reporter suggested only warning if the class both begins and ends with `:`. Is it appropriate to warn here? Is this a job best left to a static linter like Rubocop, which didn't exist at the time #1831 was opened? Or perhaps would it be better to warn only in the very specific case that #1831 was opened to address? -- https://bugs.ruby-lang.org/

Issue #21870 has been updated by duerst (Martin Dürst). Using two or more overlapping Unicode properties may not be very frequent, but in most cases isn't a mistake. If a user writes `/[\p{Word}\p{S}]/`, that expression should just match all word characters and all symbol characters, because that's most probably what the user wanted. The fact that there are some characters that are both word characters and symbol characters is irrelevant for that query, and should not produce a warning. There are many overlapping Unicode properties, because Unicode properties identify different aspects of characters (e.g. script, block, age, numeric properties,...). If we want to continue to warn about `/[:lower:]/`, that's fine, but we should warn about that specific case, not overlapping properties in general. ---------------------------------------- Bug #21870: Regexp: Warnings when using slightly overlapping \p{...} classes https://bugs.ruby-lang.org/issues/21870#change-116543 * Author: jneen (Jeanine Adkisson) * Status: Open * ruby -v: 4.0.1 * Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN, 4.0: UNKNOWN ---------------------------------------- ```ruby $VERBOSE = true # warning: character class has duplicated range: /[\p{Word}\p{S}]/ regex = /[\p{Word}\p{S}]/ ``` As far as I can tell this is a perfectly valid and non-redundant set of unicode properties, but I am still being spammed with warnings. Using `/(?:\p{Word}|\p{S})/` is kind of a workaround, but it is slower (see benchmarks below), and also less clear. They do overlap somewhat, but I think the deeper issue is there is not a convenient way to express this without falling back to raw unicode ranges. For a similar example, consider `/[\p{Word}\p{Cf}]/`, which overlap precisely on ZWJ and ZWNJ. Even with this very small overlap, Ruby issues a warning, despite neither class being removable without changing the meaning of the regexp. The regexp is valid and as far as I can tell has no practical issues - Onigmo seems to be capable of intersecting overlapping codepoint ranges. This warning was introduced back in 2009 with #1831, to help surface instances of things like `/[:lower:]/` instead of `/[[:lower:]]/`, but even then the reporter suggested only warning if the class both begins and ends with `:`. Is it appropriate to warn here? Is this a job best left to a static linter like Rubocop, which didn't exist at the time #1831 was opened? Or perhaps would it be better to warn only in the very specific case that #1831 was opened to address? -- https://bugs.ruby-lang.org/

Issue #21870 has been updated by jneen (Jeanine Adkisson). I have submitted a patch to Onigmo with the naive strategy (A) as outlined above: https://github.com/k-takata/Onigmo/pull/175 Contrary to my expectations, it does appear that `/[\p{Word}\p{Alpha}]/` and `/[a-fb-g]/` **do** continue to warn after this change, so I believe this is also a very minimal impact change. I would feel slightly better about it if I understood *why* these cases continue to warn after stripping these few lines, but at any rate this seems to be the outcome I had hoped for from the more difficult approach. Are there any other cases I should test here? ---------------------------------------- Bug #21870: Regexp: Warnings when using slightly overlapping \p{...} classes https://bugs.ruby-lang.org/issues/21870#change-116690 * Author: jneen (Jeanine Adkisson) * Status: Open * ruby -v: 4.0.1 * Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN, 4.0: UNKNOWN ---------------------------------------- ```ruby $VERBOSE = true # warning: character class has duplicated range: /[\p{Word}\p{S}]/ regex = /[\p{Word}\p{S}]/ ``` As far as I can tell this is a perfectly valid and non-redundant set of unicode properties, but I am still being spammed with warnings. Using `/(?:\p{Word}|\p{S})/` is kind of a workaround, but it is slower (see benchmarks below), and also less clear. They do overlap somewhat, but I think the deeper issue is there is not a convenient way to express this without falling back to raw unicode ranges. For a similar example, consider `/[\p{Word}\p{Cf}]/`, which overlap precisely on ZWJ and ZWNJ. Even with this very small overlap, Ruby issues a warning, despite neither class being removable without changing the meaning of the regexp. The regexp is valid and as far as I can tell has no practical issues - Onigmo seems to be capable of intersecting overlapping codepoint ranges. This warning was introduced back in 2009 with #1831, to help surface instances of things like `/[:lower:]/` instead of `/[[:lower:]]/`, but even then the reporter suggested only warning if the class both begins and ends with `:`. Is it appropriate to warn here? Is this a job best left to a static linter like Rubocop, which didn't exist at the time #1831 was opened? Or perhaps would it be better to warn only in the very specific case that #1831 was opened to address? -- https://bugs.ruby-lang.org/

Issue #21870 has been updated by jneen (Jeanine Adkisson). As a point of order - I opened a pull request against Onigmo, should that have been to Ruby instead? I'm happy to do either or both. It is a fairly simple fix, but there are some tests in Ruby main that would need to change to reflect the removal. ---------------------------------------- Bug #21870: Regexp: Warnings when using slightly overlapping \p{...} classes https://bugs.ruby-lang.org/issues/21870#change-116985 * Author: jneen (Jeanine Adkisson) * Status: Open * ruby -v: 4.0.1 * Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN, 4.0: UNKNOWN ---------------------------------------- ```ruby $VERBOSE = true # warning: character class has duplicated range: /[\p{Word}\p{S}]/ regex = /[\p{Word}\p{S}]/ ``` As far as I can tell this is a perfectly valid and non-redundant set of unicode properties, but I am still being spammed with warnings. Using `/(?:\p{Word}|\p{S})/` is kind of a workaround, but it is slower (see benchmarks below), and also less clear. They do overlap somewhat, but I think the deeper issue is there is not a convenient way to express this without falling back to raw unicode ranges. For a similar example, consider `/[\p{Word}\p{Cf}]/`, which overlap precisely on ZWJ and ZWNJ. Even with this very small overlap, Ruby issues a warning, despite neither class being removable without changing the meaning of the regexp. The regexp is valid and as far as I can tell has no practical issues - Onigmo seems to be capable of intersecting overlapping codepoint ranges. This warning was introduced back in 2009 with #1831, to help surface instances of things like `/[:lower:]/` instead of `/[[:lower:]]/`, but even then the reporter suggested only warning if the class both begins and ends with `:`. Is it appropriate to warn here? Is this a job best left to a static linter like Rubocop, which didn't exist at the time #1831 was opened? Or perhaps would it be better to warn only in the very specific case that #1831 was opened to address? -- https://bugs.ruby-lang.org/

Issue #21870 has been updated by jneen (Jeanine Adkisson). That was my expectation as well, but what I observed was that it does in fact seem to preserve warning behaviour for all of the other cases discussed here, as noted both here and in the pull request description. I would definitely feel slightly more comfortable if I could explain why, though. ---------------------------------------- Bug #21870: Regexp: Warnings when using slightly overlapping \p{...} classes https://bugs.ruby-lang.org/issues/21870#change-116988 * Author: jneen (Jeanine Adkisson) * Status: Open * ruby -v: 4.0.1 * Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN, 4.0: UNKNOWN ---------------------------------------- ```ruby $VERBOSE = true # warning: character class has duplicated range: /[\p{Word}\p{S}]/ regex = /[\p{Word}\p{S}]/ ``` As far as I can tell this is a perfectly valid and non-redundant set of unicode properties, but I am still being spammed with warnings. Using `/(?:\p{Word}|\p{S})/` is kind of a workaround, but it is slower (see benchmarks below), and also less clear. They do overlap somewhat, but I think the deeper issue is there is not a convenient way to express this without falling back to raw unicode ranges. For a similar example, consider `/[\p{Word}\p{Cf}]/`, which overlap precisely on ZWJ and ZWNJ. Even with this very small overlap, Ruby issues a warning, despite neither class being removable without changing the meaning of the regexp. The regexp is valid and as far as I can tell has no practical issues - Onigmo seems to be capable of intersecting overlapping codepoint ranges. This warning was introduced back in 2009 with #1831, to help surface instances of things like `/[:lower:]/` instead of `/[[:lower:]]/`, but even then the reporter suggested only warning if the class both begins and ends with `:`. Is it appropriate to warn here? Is this a job best left to a static linter like Rubocop, which didn't exist at the time #1831 was opened? Or perhaps would it be better to warn only in the very specific case that #1831 was opened to address? -- https://bugs.ruby-lang.org/

Issue #21870 has been updated by jneen (Jeanine Adkisson). Just wanted to double-check the status here, since we seem to have stalled out for a few months. Is there anything I could do to help move this along? ---------------------------------------- Bug #21870: Regexp: Warnings when using slightly overlapping \p{...} classes https://bugs.ruby-lang.org/issues/21870#change-118429 * Author: jneen (Jeanine Adkisson) * Status: Open * ruby -v: 4.0.1 * Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN, 4.0: UNKNOWN ---------------------------------------- ```ruby $VERBOSE = true # warning: character class has duplicated range: /[\p{Word}\p{S}]/ regex = /[\p{Word}\p{S}]/ ``` As far as I can tell this is a perfectly valid and non-redundant set of unicode properties, but I am still being spammed with warnings. Using `/(?:\p{Word}|\p{S})/` is kind of a workaround, but it is slower (see benchmarks below), and also less clear. They do overlap somewhat, but I think the deeper issue is there is not a convenient way to express this without falling back to raw unicode ranges. For a similar example, consider `/[\p{Word}\p{Cf}]/`, which overlap precisely on ZWJ and ZWNJ. Even with this very small overlap, Ruby issues a warning, despite neither class being removable without changing the meaning of the regexp. The regexp is valid and as far as I can tell has no practical issues - Onigmo seems to be capable of intersecting overlapping codepoint ranges. This warning was introduced back in 2009 with #1831, to help surface instances of things like `/[:lower:]/` instead of `/[[:lower:]]/`, but even then the reporter suggested only warning if the class both begins and ends with `:`. Is it appropriate to warn here? Is this a job best left to a static linter like Rubocop, which didn't exist at the time #1831 was opened? Or perhaps would it be better to warn only in the very specific case that #1831 was opened to address? -- https://bugs.ruby-lang.org/