[ruby-core:126900] [Ruby Feature#22400] Use rb_long_t for lengths so that String and Array can exceed 2GiB on mswin
Issue #22400 has been reported by hsbt (Hiroshi SHIBATA). ---------------------------------------- Feature #22400: Use rb_long_t for lengths so that String and Array can exceed 2GiB on mswin https://bugs.ruby-lang.org/issues/22400 * Author: hsbt (Hiroshi SHIBATA) * Status: Open ---------------------------------------- On x64-mswin64, `long` is 32 bits while pointers are 64 bits. String, Array and MatchData keep their lengths in `long`, so none of them can reach 2GiB. ```ruby "a" * 2**31 # RangeError: bignum too big to convert into 'long' File.open(path, "rb") { it.read(2**31) } # RangeError Array.new(2**31) # RangeError ``` I propose giving these lengths and indices a type of their own, `rb_long_t`, in two steps. The first step is https://github.com/ruby/ruby/pull/18862. It defines `typedef long rb_long_t` everywhere and uses it for the lengths and indices of RString, RArray and RMatch, the C API that carries them, and the core, adding `LONGT2NUM`, `NUM2LONGT` and `PRIdLONGT`. A static assert keeps it `long`, so nothing changes, and `NUM2LONG` and `FIX2LONG` still return `long`. The second step makes `rb_long_t` 64 bits on mswin, as `rb_off_t` already does for `off_t` there. mingw keeps `long`, and Fixnum (#14378) stays as is. This breaks extensions that pass a `long *` to an out-parameter such as `rb_range_beg_len`, because Ruby then writes 8 bytes into a 4-byte variable. `-we4133` makes that a compile error, and I will send fixes to the gems it hits: nokogiri, google-protobuf, nokolexbor, string_view, readline-ext and iconv. An extension supporting older Ruby can `typedef long rb_long_t` when `RB_LONGT_MAX` is undefined. No existing type fits. `ssize_t` and `intptr_t` are `int` on ILP32 and would change `long *` parameters there. `SIGNED_VALUE` is already 64 bits on mswin, so it could not land as a no-op first. To catch new code that adds a plain `long` for a length, I would like a CI job that builds with `rb_long_t` set to `long long` on Linux, where `%ld` and `long *` mismatches fail. First, do you agree with this direction and with carrying it out? If so, is `rb_long_t` a good name? And since few C extensions are distributed for mswin, I think we could make the second step in 4.1 as well, without a transition period. What do you think? -- https://bugs.ruby-lang.org/
Issue #22400 has been updated by byroot (Jean Boussier).
What do you think?
I think I'd be worth trying for a preview (there was no 4.1.0preview this year?) as to see what the blast radius really is. Other than that, I wonder if one avenue could be to have this be a build flag first, allowing gem maintainers 1 year to test and handle the change (bonus point if it's available in `ruby/setup-ruby`. ---------------------------------------- Feature #22400: Use rb_long_t for lengths so that String and Array can exceed 2GiB on mswin https://bugs.ruby-lang.org/issues/22400#change-119286 * Author: hsbt (Hiroshi SHIBATA) * Status: Open ---------------------------------------- On x64-mswin64, `long` is 32 bits while pointers are 64 bits. String, Array and MatchData keep their lengths in `long`, so none of them can reach 2GiB. ```ruby "a" * 2**31 # RangeError: bignum too big to convert into 'long' File.open(path, "rb") { it.read(2**31) } # RangeError Array.new(2**31) # RangeError ``` I propose giving these lengths and indices a type of their own, `rb_long_t`, in two steps. The first step is https://github.com/ruby/ruby/pull/18862. It defines `typedef long rb_long_t` everywhere and uses it for the lengths and indices of RString, RArray and RMatch, the C API that carries them, and the core, adding `LONGT2NUM`, `NUM2LONGT` and `PRIdLONGT`. A static assert keeps it `long`, so nothing changes, and `NUM2LONG` and `FIX2LONG` still return `long`. The second step makes `rb_long_t` 64 bits on mswin, as `rb_off_t` already does for `off_t` there. mingw keeps `long`, and Fixnum (#14378) stays as is. This breaks extensions that pass a `long *` to an out-parameter such as `rb_range_beg_len`, because Ruby then writes 8 bytes into a 4-byte variable. `-we4133` makes that a compile error, and I will send fixes to the gems it hits: nokogiri, google-protobuf, nokolexbor, string_view, readline-ext and iconv. An extension supporting older Ruby can `typedef long rb_long_t` when `RB_LONGT_MAX` is undefined. No existing type fits. `ssize_t` and `intptr_t` are `int` on ILP32 and would change `long *` parameters there. `SIGNED_VALUE` is already 64 bits on mswin, so it could not land as a no-op first. To catch new code that adds a plain `long` for a length, I would like a CI job that builds with `rb_long_t` set to `long long` on Linux, where `%ld` and `long *` mismatches fail. First, do you agree with this direction and with carrying it out? If so, is `rb_long_t` a good name? And since few C extensions are distributed for mswin, I think we could make the second step in 4.1 as well, without a transition period. What do you think? -- https://bugs.ruby-lang.org/
Issue #22400 has been updated by matz (Yukihiro Matsumoto). I agree with the direction. A 2GiB limit on a 64-bit platform should go away, and the name `rb_long_t` is fine with me since it follows `rb_off_t`. Step 1 can go into 4.1. For step 2, I am worried about switching the default in 4.1 with no trial period. Besides the `long *` out-parameters, existing extensions that do `long len = RSTRING_LEN(str)` will still compile and silently truncate. I prefer @byroot's idea: make it a build option in 4.1 and switch the default in 4.2, unless a preview shows the impact is small. Matz. ---------------------------------------- Feature #22400: Use rb_long_t for lengths so that String and Array can exceed 2GiB on mswin https://bugs.ruby-lang.org/issues/22400#change-119345 * Author: hsbt (Hiroshi SHIBATA) * Status: Open ---------------------------------------- On x64-mswin64, `long` is 32 bits while pointers are 64 bits. String, Array and MatchData keep their lengths in `long`, so none of them can reach 2GiB. ```ruby "a" * 2**31 # RangeError: bignum too big to convert into 'long' File.open(path, "rb") { it.read(2**31) } # RangeError Array.new(2**31) # RangeError ``` I propose giving these lengths and indices a type of their own, `rb_long_t`, in two steps. The first step is https://github.com/ruby/ruby/pull/18862. It defines `typedef long rb_long_t` everywhere and uses it for the lengths and indices of RString, RArray and RMatch, the C API that carries them, and the core, adding `LONGT2NUM`, `NUM2LONGT` and `PRIdLONGT`. A static assert keeps it `long`, so nothing changes, and `NUM2LONG` and `FIX2LONG` still return `long`. The second step makes `rb_long_t` 64 bits on mswin, as `rb_off_t` already does for `off_t` there. mingw keeps `long`, and Fixnum (#14378) stays as is. This breaks extensions that pass a `long *` to an out-parameter such as `rb_range_beg_len`, because Ruby then writes 8 bytes into a 4-byte variable. `-we4133` makes that a compile error, and I will send fixes to the gems it hits: nokogiri, google-protobuf, nokolexbor, string_view, readline-ext and iconv. An extension supporting older Ruby can `typedef long rb_long_t` when `RB_LONGT_MAX` is undefined. No existing type fits. `ssize_t` and `intptr_t` are `int` on ILP32 and would change `long *` parameters there. `SIGNED_VALUE` is already 64 bits on mswin, so it could not land as a no-op first. To catch new code that adds a plain `long` for a length, I would like a CI job that builds with `rb_long_t` set to `long long` on Linux, where `%ld` and `long *` mismatches fail. First, do you agree with this direction and with carrying it out? If so, is `rb_long_t` a good name? And since few C extensions are distributed for mswin, I think we could make the second step in 4.1 as well, without a transition period. What do you think? -- https://bugs.ruby-lang.org/
Issue #22400 has been updated by matz (Yukihiro Matsumoto). One more thing: as far as mswin is concerned, please feel free to go ahead as you see fit. Matz. ---------------------------------------- Feature #22400: Use rb_long_t for lengths so that String and Array can exceed 2GiB on mswin https://bugs.ruby-lang.org/issues/22400#change-119349 * Author: hsbt (Hiroshi SHIBATA) * Status: Open ---------------------------------------- On x64-mswin64, `long` is 32 bits while pointers are 64 bits. String, Array and MatchData keep their lengths in `long`, so none of them can reach 2GiB. ```ruby "a" * 2**31 # RangeError: bignum too big to convert into 'long' File.open(path, "rb") { it.read(2**31) } # RangeError Array.new(2**31) # RangeError ``` I propose giving these lengths and indices a type of their own, `rb_long_t`, in two steps. The first step is https://github.com/ruby/ruby/pull/18862. It defines `typedef long rb_long_t` everywhere and uses it for the lengths and indices of RString, RArray and RMatch, the C API that carries them, and the core, adding `LONGT2NUM`, `NUM2LONGT` and `PRIdLONGT`. A static assert keeps it `long`, so nothing changes, and `NUM2LONG` and `FIX2LONG` still return `long`. The second step makes `rb_long_t` 64 bits on mswin, as `rb_off_t` already does for `off_t` there. mingw keeps `long`, and Fixnum (#14378) stays as is. This breaks extensions that pass a `long *` to an out-parameter such as `rb_range_beg_len`, because Ruby then writes 8 bytes into a 4-byte variable. `-we4133` makes that a compile error, and I will send fixes to the gems it hits: nokogiri, google-protobuf, nokolexbor, string_view, readline-ext and iconv. An extension supporting older Ruby can `typedef long rb_long_t` when `RB_LONGT_MAX` is undefined. No existing type fits. `ssize_t` and `intptr_t` are `int` on ILP32 and would change `long *` parameters there. `SIGNED_VALUE` is already 64 bits on mswin, so it could not land as a no-op first. To catch new code that adds a plain `long` for a length, I would like a CI job that builds with `rb_long_t` set to `long long` on Linux, where `%ld` and `long *` mismatches fail. First, do you agree with this direction and with carrying it out? If so, is `rb_long_t` a good name? And since few C extensions are distributed for mswin, I think we could make the second step in 4.1 as well, without a transition period. What do you think? -- https://bugs.ruby-lang.org/
Issue #22400 has been updated by rhenium (Kazuki Yamaguchi). I think this is a good step, but the conversion macro `NUM2LONG(val)` vs `NUM2LONGT(val)` seems incredibly confusing. Were completely different names considered, such as `len`? ---------------------------------------- Feature #22400: Use rb_long_t for lengths so that String and Array can exceed 2GiB on mswin https://bugs.ruby-lang.org/issues/22400#change-119353 * Author: hsbt (Hiroshi SHIBATA) * Status: Open ---------------------------------------- On x64-mswin64, `long` is 32 bits while pointers are 64 bits. String, Array and MatchData keep their lengths in `long`, so none of them can reach 2GiB. ```ruby "a" * 2**31 # RangeError: bignum too big to convert into 'long' File.open(path, "rb") { it.read(2**31) } # RangeError Array.new(2**31) # RangeError ``` I propose giving these lengths and indices a type of their own, `rb_long_t`, in two steps. The first step is https://github.com/ruby/ruby/pull/18862. It defines `typedef long rb_long_t` everywhere and uses it for the lengths and indices of RString, RArray and RMatch, the C API that carries them, and the core, adding `LONGT2NUM`, `NUM2LONGT` and `PRIdLONGT`. A static assert keeps it `long`, so nothing changes, and `NUM2LONG` and `FIX2LONG` still return `long`. The second step makes `rb_long_t` 64 bits on mswin, as `rb_off_t` already does for `off_t` there. mingw keeps `long`, and Fixnum (#14378) stays as is. This breaks extensions that pass a `long *` to an out-parameter such as `rb_range_beg_len`, because Ruby then writes 8 bytes into a 4-byte variable. `-we4133` makes that a compile error, and I will send fixes to the gems it hits: nokogiri, google-protobuf, nokolexbor, string_view, readline-ext and iconv. An extension supporting older Ruby can `typedef long rb_long_t` when `RB_LONGT_MAX` is undefined. No existing type fits. `ssize_t` and `intptr_t` are `int` on ILP32 and would change `long *` parameters there. `SIGNED_VALUE` is already 64 bits on mswin, so it could not land as a no-op first. To catch new code that adds a plain `long` for a length, I would like a CI job that builds with `rb_long_t` set to `long long` on Linux, where `%ld` and `long *` mismatches fail. First, do you agree with this direction and with carrying it out? If so, is `rb_long_t` a good name? And since few C extensions are distributed for mswin, I think we could make the second step in 4.1 as well, without a transition period. What do you think? -- https://bugs.ruby-lang.org/
Issue #22400 has been updated by byroot (Jean Boussier).
Were completely different names considered, such as len?
Also the natural type to go for would be `size_t`, so `rb_size_t` would make more sense than `rb_long_t`? But then there's the sign difference... ---------------------------------------- Feature #22400: Use rb_long_t for lengths so that String and Array can exceed 2GiB on mswin https://bugs.ruby-lang.org/issues/22400#change-119355 * Author: hsbt (Hiroshi SHIBATA) * Status: Open ---------------------------------------- On x64-mswin64, `long` is 32 bits while pointers are 64 bits. String, Array and MatchData keep their lengths in `long`, so none of them can reach 2GiB. ```ruby "a" * 2**31 # RangeError: bignum too big to convert into 'long' File.open(path, "rb") { it.read(2**31) } # RangeError Array.new(2**31) # RangeError ``` I propose giving these lengths and indices a type of their own, `rb_long_t`, in two steps. The first step is https://github.com/ruby/ruby/pull/18862. It defines `typedef long rb_long_t` everywhere and uses it for the lengths and indices of RString, RArray and RMatch, the C API that carries them, and the core, adding `LONGT2NUM`, `NUM2LONGT` and `PRIdLONGT`. A static assert keeps it `long`, so nothing changes, and `NUM2LONG` and `FIX2LONG` still return `long`. The second step makes `rb_long_t` 64 bits on mswin, as `rb_off_t` already does for `off_t` there. mingw keeps `long`, and Fixnum (#14378) stays as is. This breaks extensions that pass a `long *` to an out-parameter such as `rb_range_beg_len`, because Ruby then writes 8 bytes into a 4-byte variable. `-we4133` makes that a compile error, and I will send fixes to the gems it hits: nokogiri, google-protobuf, nokolexbor, string_view, readline-ext and iconv. An extension supporting older Ruby can `typedef long rb_long_t` when `RB_LONGT_MAX` is undefined. No existing type fits. `ssize_t` and `intptr_t` are `int` on ILP32 and would change `long *` parameters there. `SIGNED_VALUE` is already 64 bits on mswin, so it could not land as a no-op first. To catch new code that adds a plain `long` for a length, I would like a CI job that builds with `rb_long_t` set to `long long` on Linux, where `%ld` and `long *` mismatches fail. First, do you agree with this direction and with carrying it out? If so, is `rb_long_t` a good name? And since few C extensions are distributed for mswin, I think we could make the second step in 4.1 as well, without a transition period. What do you think? -- https://bugs.ruby-lang.org/
Issue #22400 has been updated by hsbt (Hiroshi SHIBATA). Thank you both. I'd still like to keep `rb_long_t`, since it replaces `long` one for one, like `rb_off_t` and `OFFT2NUM`. `rb_size_t` would sit next to `SIZET2NUM` for `size_t`, which seems even easier to confuse. ---------------------------------------- Feature #22400: Use rb_long_t for lengths so that String and Array can exceed 2GiB on mswin https://bugs.ruby-lang.org/issues/22400#change-119362 * Author: hsbt (Hiroshi SHIBATA) * Status: Open ---------------------------------------- On x64-mswin64, `long` is 32 bits while pointers are 64 bits. String, Array and MatchData keep their lengths in `long`, so none of them can reach 2GiB. ```ruby "a" * 2**31 # RangeError: bignum too big to convert into 'long' File.open(path, "rb") { it.read(2**31) } # RangeError Array.new(2**31) # RangeError ``` I propose giving these lengths and indices a type of their own, `rb_long_t`, in two steps. The first step is https://github.com/ruby/ruby/pull/18862. It defines `typedef long rb_long_t` everywhere and uses it for the lengths and indices of RString, RArray and RMatch, the C API that carries them, and the core, adding `LONGT2NUM`, `NUM2LONGT` and `PRIdLONGT`. A static assert keeps it `long`, so nothing changes, and `NUM2LONG` and `FIX2LONG` still return `long`. The second step makes `rb_long_t` 64 bits on mswin, as `rb_off_t` already does for `off_t` there. mingw keeps `long`, and Fixnum (#14378) stays as is. This breaks extensions that pass a `long *` to an out-parameter such as `rb_range_beg_len`, because Ruby then writes 8 bytes into a 4-byte variable. `-we4133` makes that a compile error, and I will send fixes to the gems it hits: nokogiri, google-protobuf, nokolexbor, string_view, readline-ext and iconv. An extension supporting older Ruby can `typedef long rb_long_t` when `RB_LONGT_MAX` is undefined. No existing type fits. `ssize_t` and `intptr_t` are `int` on ILP32 and would change `long *` parameters there. `SIGNED_VALUE` is already 64 bits on mswin, so it could not land as a no-op first. To catch new code that adds a plain `long` for a length, I would like a CI job that builds with `rb_long_t` set to `long long` on Linux, where `%ld` and `long *` mismatches fail. First, do you agree with this direction and with carrying it out? If so, is `rb_long_t` a good name? And since few C extensions are distributed for mswin, I think we could make the second step in 4.1 as well, without a transition period. What do you think? -- https://bugs.ruby-lang.org/
Issue #22400 has been updated by rhenium (Kazuki Yamaguchi). My suggestion would be `rb_len_t`, or possibly index or count instead of len. I don't think this is really comparable to `rb_off_t` ([Bug #5317]), which appears to have been introduced as an equivalent of POSIX-like `off_t` on Windows. Besides LONG and LONGT seem too similar, I think naming the new type after `long` is misleading when the plan is to eventually decouple it from `long`. Since we're going to expect extensions to eventually use this new type and the helpers, why not give it a more descriptive name? ---------------------------------------- Feature #22400: Use rb_long_t for lengths so that String and Array can exceed 2GiB on mswin https://bugs.ruby-lang.org/issues/22400#change-119380 * Author: hsbt (Hiroshi SHIBATA) * Status: Open ---------------------------------------- On x64-mswin64, `long` is 32 bits while pointers are 64 bits. String, Array and MatchData keep their lengths in `long`, so none of them can reach 2GiB. ```ruby "a" * 2**31 # RangeError: bignum too big to convert into 'long' File.open(path, "rb") { it.read(2**31) } # RangeError Array.new(2**31) # RangeError ``` I propose giving these lengths and indices a type of their own, `rb_long_t`, in two steps. The first step is https://github.com/ruby/ruby/pull/18862. It defines `typedef long rb_long_t` everywhere and uses it for the lengths and indices of RString, RArray and RMatch, the C API that carries them, and the core, adding `LONGT2NUM`, `NUM2LONGT` and `PRIdLONGT`. A static assert keeps it `long`, so nothing changes, and `NUM2LONG` and `FIX2LONG` still return `long`. The second step makes `rb_long_t` 64 bits on mswin, as `rb_off_t` already does for `off_t` there. mingw keeps `long`, and Fixnum (#14378) stays as is. This breaks extensions that pass a `long *` to an out-parameter such as `rb_range_beg_len`, because Ruby then writes 8 bytes into a 4-byte variable. `-we4133` makes that a compile error, and I will send fixes to the gems it hits: nokogiri, google-protobuf, nokolexbor, string_view, readline-ext and iconv. An extension supporting older Ruby can `typedef long rb_long_t` when `RB_LONGT_MAX` is undefined. No existing type fits. `ssize_t` and `intptr_t` are `int` on ILP32 and would change `long *` parameters there. `SIGNED_VALUE` is already 64 bits on mswin, so it could not land as a no-op first. To catch new code that adds a plain `long` for a length, I would like a CI job that builds with `rb_long_t` set to `long long` on Linux, where `%ld` and `long *` mismatches fail. First, do you agree with this direction and with carrying it out? If so, is `rb_long_t` a good name? And since few C extensions are distributed for mswin, I think we could make the second step in 4.1 as well, without a transition period. What do you think? -- https://bugs.ruby-lang.org/
Issue #22400 has been updated by hsbt (Hiroshi SHIBATA). Thanks. Your comment reminded me of what matz said at the developers' meeting: the representation should say 64 bits instead of relying on `long` or `long long`, whose meaning can change. So this type may well stop being `long` someday, and I agree a name based on `long` would mislead. I've renamed it to `rb_len_t` in https://github.com/ruby/ruby/pull/18862, with `LEN2NUM`, `NUM2LEN` and `PRIdLEN`. ---------------------------------------- Feature #22400: Use rb_long_t for lengths so that String and Array can exceed 2GiB on mswin https://bugs.ruby-lang.org/issues/22400#change-119393 * Author: hsbt (Hiroshi SHIBATA) * Status: Open ---------------------------------------- On x64-mswin64, `long` is 32 bits while pointers are 64 bits. String, Array and MatchData keep their lengths in `long`, so none of them can reach 2GiB. ```ruby "a" * 2**31 # RangeError: bignum too big to convert into 'long' File.open(path, "rb") { it.read(2**31) } # RangeError Array.new(2**31) # RangeError ``` I propose giving these lengths and indices a type of their own, `rb_long_t`, in two steps. The first step is https://github.com/ruby/ruby/pull/18862. It defines `typedef long rb_long_t` everywhere and uses it for the lengths and indices of RString, RArray and RMatch, the C API that carries them, and the core, adding `LONGT2NUM`, `NUM2LONGT` and `PRIdLONGT`. A static assert keeps it `long`, so nothing changes, and `NUM2LONG` and `FIX2LONG` still return `long`. The second step makes `rb_long_t` 64 bits on mswin, as `rb_off_t` already does for `off_t` there. mingw keeps `long`, and Fixnum (#14378) stays as is. This breaks extensions that pass a `long *` to an out-parameter such as `rb_range_beg_len`, because Ruby then writes 8 bytes into a 4-byte variable. `-we4133` makes that a compile error, and I will send fixes to the gems it hits: nokogiri, google-protobuf, nokolexbor, string_view, readline-ext and iconv. An extension supporting older Ruby can `typedef long rb_long_t` when `RB_LONGT_MAX` is undefined. No existing type fits. `ssize_t` and `intptr_t` are `int` on ILP32 and would change `long *` parameters there. `SIGNED_VALUE` is already 64 bits on mswin, so it could not land as a no-op first. To catch new code that adds a plain `long` for a length, I would like a CI job that builds with `rb_long_t` set to `long long` on Linux, where `%ld` and `long *` mismatches fail. First, do you agree with this direction and with carrying it out? If so, is `rb_long_t` a good name? And since few C extensions are distributed for mswin, I think we could make the second step in 4.1 as well, without a transition period. What do you think? -- https://bugs.ruby-lang.org/
Issue #22400 has been updated by byroot (Jean Boussier). @hsbt what's the migration path for C extensions? I don't see a `HAVE_RB_LEN_T`, should we provide one so that extensions can declare: ```c #ifndef HAVE_RB_LEN_T typedef long rb_len_t; typedef unsigned long rb_ulen_t; #endif ``` ---------------------------------------- Feature #22400: Use rb_long_t for lengths so that String and Array can exceed 2GiB on mswin https://bugs.ruby-lang.org/issues/22400#change-119422 * Author: hsbt (Hiroshi SHIBATA) * Status: Open ---------------------------------------- On x64-mswin64, `long` is 32 bits while pointers are 64 bits. String, Array and MatchData keep their lengths in `long`, so none of them can reach 2GiB. ```ruby "a" * 2**31 # RangeError: bignum too big to convert into 'long' File.open(path, "rb") { it.read(2**31) } # RangeError Array.new(2**31) # RangeError ``` I propose giving these lengths and indices a type of their own, `rb_long_t`, in two steps. The first step is https://github.com/ruby/ruby/pull/18862. It defines `typedef long rb_long_t` everywhere and uses it for the lengths and indices of RString, RArray and RMatch, the C API that carries them, and the core, adding `LONGT2NUM`, `NUM2LONGT` and `PRIdLONGT`. A static assert keeps it `long`, so nothing changes, and `NUM2LONG` and `FIX2LONG` still return `long`. The second step makes `rb_long_t` 64 bits on mswin, as `rb_off_t` already does for `off_t` there. mingw keeps `long`, and Fixnum (#14378) stays as is. This breaks extensions that pass a `long *` to an out-parameter such as `rb_range_beg_len`, because Ruby then writes 8 bytes into a 4-byte variable. `-we4133` makes that a compile error, and I will send fixes to the gems it hits: nokogiri, google-protobuf, nokolexbor, string_view, readline-ext and iconv. An extension supporting older Ruby can `typedef long rb_long_t` when `RB_LONGT_MAX` is undefined. No existing type fits. `ssize_t` and `intptr_t` are `int` on ILP32 and would change `long *` parameters there. `SIGNED_VALUE` is already 64 bits on mswin, so it could not land as a no-op first. To catch new code that adds a plain `long` for a length, I would like a CI job that builds with `rb_long_t` set to `long long` on Linux, where `%ld` and `long *` mismatches fail. First, do you agree with this direction and with carrying it out? If so, is `rb_long_t` a good name? And since few C extensions are distributed for mswin, I think we could make the second step in 4.1 as well, without a transition period. What do you think? -- https://bugs.ruby-lang.org/
Issue #22400 has been updated by hsbt (Hiroshi SHIBATA). `#ifndef SIZEOF_RB_LEN_T` works for that today, since it is defined together with `rb_len_t`. Still, I agree a `HAVE_` style macro like `HAVE_RB_IO_T` is the usual convention and easier to find, so I'll add `HAVE_RB_LEN_T`. ---------------------------------------- Feature #22400: Use rb_long_t for lengths so that String and Array can exceed 2GiB on mswin https://bugs.ruby-lang.org/issues/22400#change-119425 * Author: hsbt (Hiroshi SHIBATA) * Status: Open ---------------------------------------- On x64-mswin64, `long` is 32 bits while pointers are 64 bits. String, Array and MatchData keep their lengths in `long`, so none of them can reach 2GiB. ```ruby "a" * 2**31 # RangeError: bignum too big to convert into 'long' File.open(path, "rb") { it.read(2**31) } # RangeError Array.new(2**31) # RangeError ``` I propose giving these lengths and indices a type of their own, `rb_long_t`, in two steps. The first step is https://github.com/ruby/ruby/pull/18862. It defines `typedef long rb_long_t` everywhere and uses it for the lengths and indices of RString, RArray and RMatch, the C API that carries them, and the core, adding `LONGT2NUM`, `NUM2LONGT` and `PRIdLONGT`. A static assert keeps it `long`, so nothing changes, and `NUM2LONG` and `FIX2LONG` still return `long`. The second step makes `rb_long_t` 64 bits on mswin, as `rb_off_t` already does for `off_t` there. mingw keeps `long`, and Fixnum (#14378) stays as is. This breaks extensions that pass a `long *` to an out-parameter such as `rb_range_beg_len`, because Ruby then writes 8 bytes into a 4-byte variable. `-we4133` makes that a compile error, and I will send fixes to the gems it hits: nokogiri, google-protobuf, nokolexbor, string_view, readline-ext and iconv. An extension supporting older Ruby can `typedef long rb_long_t` when `RB_LONGT_MAX` is undefined. No existing type fits. `ssize_t` and `intptr_t` are `int` on ILP32 and would change `long *` parameters there. `SIGNED_VALUE` is already 64 bits on mswin, so it could not land as a no-op first. To catch new code that adds a plain `long` for a length, I would like a CI job that builds with `rb_long_t` set to `long long` on Linux, where `%ld` and `long *` mismatches fail. First, do you agree with this direction and with carrying it out? If so, is `rb_long_t` a good name? And since few C extensions are distributed for mswin, I think we could make the second step in 4.1 as well, without a transition period. What do you think? -- https://bugs.ruby-lang.org/
Issue #22400 has been updated by larskanis (Lars Kanis). Both MSWIN and MINGW intentionally use the same ABI. They only have a different toolset. So I don't think we should introduce a ABI difference on the ruby C interface for them. Likewise there's no good reason to handle MINGW and MSWIN differently because of compatibility, IMHO. The same source gems are used for MSWIN and for MINGW. So they need to be changed regarding `rb_len_t` in order to be compatible to MSWIN anyway. I think this is manageable for the time frame until Ruby-4.2 comes out, so that they are compatible to 64 bit `rb_len_t` on both MSWIN and MINGW then. There are currently more binary gems available for MINGW than for MSWIN. But it doesn't matter, since already released binaries are bound to the Ruby ABI 3.3, 3.4, 4.0, etc. and Ruby-4.1/4.2 need new binaries anyway. I understand that the ruby core developers prefer MSWIN and that testing on MINGW is cumbersome for you and slows down development. So I'm fine with checking the MSWIN related changes and port them to MINGW where necessary. You could give me a hint to do so, if there's some need. Anyway it's great that the Windows port receives some attention recently and I really appreciate all the efforts! ---------------------------------------- Feature #22400: Use rb_long_t for lengths so that String and Array can exceed 2GiB on mswin https://bugs.ruby-lang.org/issues/22400#change-119565 * Author: hsbt (Hiroshi SHIBATA) * Status: Open ---------------------------------------- On x64-mswin64, `long` is 32 bits while pointers are 64 bits. String, Array and MatchData keep their lengths in `long`, so none of them can reach 2GiB. ```ruby "a" * 2**31 # RangeError: bignum too big to convert into 'long' File.open(path, "rb") { it.read(2**31) } # RangeError Array.new(2**31) # RangeError ``` I propose giving these lengths and indices a type of their own, `rb_long_t`, in two steps. The first step is https://github.com/ruby/ruby/pull/18862. It defines `typedef long rb_long_t` everywhere and uses it for the lengths and indices of RString, RArray and RMatch, the C API that carries them, and the core, adding `LONGT2NUM`, `NUM2LONGT` and `PRIdLONGT`. A static assert keeps it `long`, so nothing changes, and `NUM2LONG` and `FIX2LONG` still return `long`. The second step makes `rb_long_t` 64 bits on mswin, as `rb_off_t` already does for `off_t` there. mingw keeps `long`, and Fixnum (#14378) stays as is. This breaks extensions that pass a `long *` to an out-parameter such as `rb_range_beg_len`, because Ruby then writes 8 bytes into a 4-byte variable. `-we4133` makes that a compile error, and I will send fixes to the gems it hits: nokogiri, google-protobuf, nokolexbor, string_view, readline-ext and iconv. An extension supporting older Ruby can `typedef long rb_long_t` when `RB_LONGT_MAX` is undefined. No existing type fits. `ssize_t` and `intptr_t` are `int` on ILP32 and would change `long *` parameters there. `SIGNED_VALUE` is already 64 bits on mswin, so it could not land as a no-op first. To catch new code that adds a plain `long` for a length, I would like a CI job that builds with `rb_long_t` set to `long long` on Linux, where `%ld` and `long *` mismatches fail. First, do you agree with this direction and with carrying it out? If so, is `rb_long_t` a good name? And since few C extensions are distributed for mswin, I think we could make the second step in 4.1 as well, without a transition period. What do you think? -- https://bugs.ruby-lang.org/
Issue #22400 has been updated by hsbt (Hiroshi SHIBATA). Status changed from Open to Closed Step 2 is merged in https://github.com/ruby/ruby/pull/19312, so String, Array and MatchData can now be 2GiB or longer on mswin. Next I'll work toward a 64-bit Fixnum on mswin (#14378) step by step, targeting Ruby 4.2. ---------------------------------------- Feature #22400: Use rb_long_t for lengths so that String and Array can exceed 2GiB on mswin https://bugs.ruby-lang.org/issues/22400#change-119576 * Author: hsbt (Hiroshi SHIBATA) * Status: Closed ---------------------------------------- On x64-mswin64, `long` is 32 bits while pointers are 64 bits. String, Array and MatchData keep their lengths in `long`, so none of them can reach 2GiB. ```ruby "a" * 2**31 # RangeError: bignum too big to convert into 'long' File.open(path, "rb") { it.read(2**31) } # RangeError Array.new(2**31) # RangeError ``` I propose giving these lengths and indices a type of their own, `rb_long_t`, in two steps. The first step is https://github.com/ruby/ruby/pull/18862. It defines `typedef long rb_long_t` everywhere and uses it for the lengths and indices of RString, RArray and RMatch, the C API that carries them, and the core, adding `LONGT2NUM`, `NUM2LONGT` and `PRIdLONGT`. A static assert keeps it `long`, so nothing changes, and `NUM2LONG` and `FIX2LONG` still return `long`. The second step makes `rb_long_t` 64 bits on mswin, as `rb_off_t` already does for `off_t` there. mingw keeps `long`, and Fixnum (#14378) stays as is. This breaks extensions that pass a `long *` to an out-parameter such as `rb_range_beg_len`, because Ruby then writes 8 bytes into a 4-byte variable. `-we4133` makes that a compile error, and I will send fixes to the gems it hits: nokogiri, google-protobuf, nokolexbor, string_view, readline-ext and iconv. An extension supporting older Ruby can `typedef long rb_long_t` when `RB_LONGT_MAX` is undefined. No existing type fits. `ssize_t` and `intptr_t` are `int` on ILP32 and would change `long *` parameters there. `SIGNED_VALUE` is already 64 bits on mswin, so it could not land as a no-op first. To catch new code that adds a plain `long` for a length, I would like a CI job that builds with `rb_long_t` set to `long long` on Linux, where `%ld` and `long *` mismatches fail. First, do you agree with this direction and with carrying it out? If so, is `rb_long_t` a good name? And since few C extensions are distributed for mswin, I think we could make the second step in 4.1 as well, without a transition period. What do you think? -- https://bugs.ruby-lang.org/
participants (5)
-
byroot (Jean Boussier) -
hsbt (Hiroshi SHIBATA)
-
larskanis (Lars Kanis)
-
matz (Yukihiro Matsumoto)
-
rhenium (Kazuki Yamaguchi)