Issue #19908 has been updated by duerst (Martin Dürst). There is a serious issue than just whether using an '_' or an '=' in the property: Unicode 15.1 makes some serious changes to grapheme clusters. Our implementation (function 'node_extended_grapheme_cluster' in regparse.c) is based on Unicode 11.0, in particular https://www.unicode.org/reports/tr29/tr29-33.html#Grapheme_Cluster_Boundarie.... This is quite a bit different from the current version at https://www.unicode.org/reports/tr29/tr29-43.html#Grapheme_Cluster_Boundarie.... One major difference is that for Unicode 11.0, there was a regular expression for grapheme clusters, which I just implemented in the above function. Unicode 15.1 just says that it's possible to use a regular expression, but doesn't give this regular expression. From reading through https://www.unicode.org/versions/Unicode15.1.0/#Migration, that's the main issue affecting Ruby. ---------------------------------------- Feature #19908: Update to Unicode 15.1 https://bugs.ruby-lang.org/issues/19908#change-105854 * Author: nobu (Nobuyoshi Nakada) * Status: Assigned * Priority: Normal * Assignee: duerst (Martin Dürst) ---------------------------------------- The Unicode 15.1 is released. The current enc-unicode.rb seems to fail because of `Indic_Conjunct_break` properties with values. I'm not sure how these properties should be handled well. `/\p{InCB_Liner}/` or `/\p{InCB=Liner}/` as the comments in that file? https://github.com/nobu/ruby/tree/unicode-15.1 is the former. -- https://bugs.ruby-lang.org/

Issue #19908 has been updated by janosch-x (Janosch Müller). Is not [this](https://www.unicode.org/reports/tr29/tr29-43.html#Regex_Definitions) the updated regular expression? ```diff ccs-base := [\p{L}\p{N}\p{P}\p{S}\p{Zs}] ccs-extend := [\p{M}\p{Join_Control}] extended_base := ccs-base | hangul-syllable -crlf := CR LF +crlf := CR LF | CR | LF legacy-core := hangul-syllable | ri-sequence | xpicto-sequence legacy-postcore := [Extend ZWJ] core := hangul-syllable | ri-sequence | xpicto-sequence +| conjunctCluster | [^Control CR LF] postcore := [Extend ZWJ SpacingMark] precore := Prepend hangul-syllable := L* (V+ | LV V* | LVT) T* | L+ | T+ xpicto-sequence := \p{Extended_Pictographic} (Extend* ZWJ \p{Extended_Pictographic})* +conjunctCluster := \p{InCB=Consonant} ([\p{InCB=Extend} \p{InCB=Linker}]* \p{InCB=Linker} [\p{InCB=Extend} \p{InCB=Linker}]* \p{InCB=Consonant})+ ``` ---------------------------------------- Feature #19908: Update to Unicode 15.1 https://bugs.ruby-lang.org/issues/19908#change-106054 * Author: nobu (Nobuyoshi Nakada) * Status: Assigned * Priority: Normal * Assignee: duerst (Martin Dürst) ---------------------------------------- The Unicode 15.1 is released. The current enc-unicode.rb seems to fail because of `Indic_Conjunct_break` properties with values. I'm not sure how these properties should be handled well. `/\p{InCB_Liner}/` or `/\p{InCB=Liner}/` as the comments in that file? https://github.com/nobu/ruby/tree/unicode-15.1 is the former. -- https://bugs.ruby-lang.org/

Issue #19908 has been updated by hsbt (Hiroshi SHIBATA). Unicode 16.0 has been released. https://www.unicode.org/versions/Unicode16.0.0/ Should we move this instead of 15.1? ---------------------------------------- Feature #19908: Update to Unicode 15.1 https://bugs.ruby-lang.org/issues/19908#change-109722 * Author: nobu (Nobuyoshi Nakada) * Status: Assigned * Assignee: duerst (Martin Dürst) ---------------------------------------- The Unicode 15.1 is released. The current enc-unicode.rb seems to fail because of `Indic_Conjunct_break` properties with values. I'm not sure how these properties should be handled well. `/\p{InCB_Liner}/` or `/\p{InCB=Liner}/` as the comments in that file? https://github.com/nobu/ruby/tree/unicode-15.1 is the former. -- https://bugs.ruby-lang.org/

Issue #19908 has been updated by hsbt (Hiroshi SHIBATA).

I think it's more prudent to do 15.1 first, then 16.0.

Agreed, thanks! ---------------------------------------- Feature #19908: Update to Unicode 15.1 https://bugs.ruby-lang.org/issues/19908#change-109726 * Author: nobu (Nobuyoshi Nakada) * Status: Assigned * Assignee: duerst (Martin Dürst) ---------------------------------------- The Unicode 15.1 is released. The current enc-unicode.rb seems to fail because of `Indic_Conjunct_break` properties with values. I'm not sure how these properties should be handled well. `/\p{InCB_Liner}/` or `/\p{InCB=Liner}/` as the comments in that file? https://github.com/nobu/ruby/tree/unicode-15.1 is the former. -- https://bugs.ruby-lang.org/

Issue #19908 has been updated by mame (Yusuke Endoh). @duerst What do you think? ---------------------------------------- Feature #19908: Update to Unicode 15.1 https://bugs.ruby-lang.org/issues/19908#change-112242 * Author: nobu (Nobuyoshi Nakada) * Status: Assigned * Assignee: duerst (Martin Dürst) ---------------------------------------- The Unicode 15.1 is released. The current enc-unicode.rb seems to fail because of `Indic_Conjunct_break` properties with values. I'm not sure how these properties should be handled well. `/\p{InCB_Liner}/` or `/\p{InCB=Liner}/` as the comments in that file? https://github.com/nobu/ruby/tree/unicode-15.1 is the former. -- https://bugs.ruby-lang.org/

Issue #19908 has been updated by naruse (Yui NARUSE). The change looks good to me. Since you have already contributed reline and show your engineering skill, and now you also want to contribute to ruby/ruby, I think you should have commit right for ruby/ruby and commit this change by yourself. @matz How do you think? ---------------------------------------- Feature #19908: Update to Unicode 15.1 https://bugs.ruby-lang.org/issues/19908#change-112335 * Author: nobu (Nobuyoshi Nakada) * Status: Assigned * Assignee: duerst (Martin Dürst) ---------------------------------------- The Unicode 15.1 is released. The current enc-unicode.rb seems to fail because of `Indic_Conjunct_break` properties with values. I'm not sure how these properties should be handled well. `/\p{InCB_Liner}/` or `/\p{InCB=Liner}/` as the comments in that file? https://github.com/nobu/ruby/tree/unicode-15.1 is the former. -- https://bugs.ruby-lang.org/

Issue #19908 has been updated by mame (Yusuke Endoh). I'd also like to introduce ima1zumi-san as a candidate for committer. She has been actively working on irb and reline, has deep knowledge and a strong interest in character encoding, and is highly recognized, as she was endorsed by @naruse, the maintainer of Ruby's encoding system. With her contributions extending towards Ruby itself, I support her nomination. ---------------------------------------- Feature #19908: Update to Unicode 15.1 https://bugs.ruby-lang.org/issues/19908#change-112337 * Author: nobu (Nobuyoshi Nakada) * Status: Assigned * Assignee: duerst (Martin Dürst) ---------------------------------------- The Unicode 15.1 is released. The current enc-unicode.rb seems to fail because of `Indic_Conjunct_break` properties with values. I'm not sure how these properties should be handled well. `/\p{InCB_Liner}/` or `/\p{InCB=Liner}/` as the comments in that file? https://github.com/nobu/ruby/tree/unicode-15.1 is the former. -- https://bugs.ruby-lang.org/

Issue #19908 has been updated by k0kubun (Takashi Kokubun). +1 ---------------------------------------- Feature #19908: Update to Unicode 15.1 https://bugs.ruby-lang.org/issues/19908#change-112339 * Author: nobu (Nobuyoshi Nakada) * Status: Assigned * Assignee: duerst (Martin Dürst) ---------------------------------------- The Unicode 15.1 is released. The current enc-unicode.rb seems to fail because of `Indic_Conjunct_break` properties with values. I'm not sure how these properties should be handled well. `/\p{InCB_Liner}/` or `/\p{InCB=Liner}/` as the comments in that file? https://github.com/nobu/ruby/tree/unicode-15.1 is the former. -- https://bugs.ruby-lang.org/

Issue #19908 has been updated by mrkn (Kenta Murata). +1 ---------------------------------------- Feature #19908: Update to Unicode 15.1 https://bugs.ruby-lang.org/issues/19908#change-112341 * Author: nobu (Nobuyoshi Nakada) * Status: Assigned * Assignee: duerst (Martin Dürst) ---------------------------------------- The Unicode 15.1 is released. The current enc-unicode.rb seems to fail because of `Indic_Conjunct_break` properties with values. I'm not sure how these properties should be handled well. `/\p{InCB_Liner}/` or `/\p{InCB=Liner}/` as the comments in that file? https://github.com/nobu/ruby/tree/unicode-15.1 is the former. -- https://bugs.ruby-lang.org/

Issue #19908 has been updated by matz (Yukihiro Matsumoto). #note-16 Approved. Matz. ---------------------------------------- Feature #19908: Update to Unicode 15.1 https://bugs.ruby-lang.org/issues/19908#change-112356 * Author: nobu (Nobuyoshi Nakada) * Status: Assigned * Assignee: duerst (Martin Dürst) ---------------------------------------- The Unicode 15.1 is released. The current enc-unicode.rb seems to fail because of `Indic_Conjunct_break` properties with values. I'm not sure how these properties should be handled well. `/\p{InCB_Liner}/` or `/\p{InCB=Liner}/` as the comments in that file? https://github.com/nobu/ruby/tree/unicode-15.1 is the former. -- https://bugs.ruby-lang.org/

Issue #19908 has been updated by ima1zumi (Mari Imaizumi). @hsbt I've sent an email to cvs-admin and opened https://github.com/ruby/git.ruby-lang.org/pull/91 ---------------------------------------- Feature #19908: Update to Unicode 15.1 https://bugs.ruby-lang.org/issues/19908#change-112358 * Author: nobu (Nobuyoshi Nakada) * Status: Assigned * Assignee: duerst (Martin Dürst) ---------------------------------------- The Unicode 15.1 is released. The current enc-unicode.rb seems to fail because of `Indic_Conjunct_break` properties with values. I'm not sure how these properties should be handled well. `/\p{InCB_Liner}/` or `/\p{InCB=Liner}/` as the comments in that file? https://github.com/nobu/ruby/tree/unicode-15.1 is the former. -- https://bugs.ruby-lang.org/