October 2024 - ruby-core - ml.ruby-lang.org

[ruby-core:119556] [Ruby master Bug#20804] Stop reserving stack ahead-of-time in on Linux
by kjtsanaktsidis (KJ Tsanaktsidis) 21 Oct '24

21 Oct '24

Issue #20804 has been reported by kjtsanaktsidis (KJ Tsanaktsidis). ---------------------------------------- Bug #20804: Stop reserving stack ahead-of-time in on Linux https://bugs.ruby-lang.org/issues/20804 * Author: kjtsanaktsidis (KJ Tsanaktsidis) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- In Linux, the main thread generally only gets a small stack mapped in initially. As the application attempts to use more stack memory, the kernel will map in more stack pages. In https://github.com/ruby/ruby/pull/822, we added some logic to force the kernel to eagerly map and fault in the entire stack by writing a fake array near the bottom. This was done in order to fix some cases where heap memory was unexpectedly being allocated in locations close to the stack, which then prevented the stack from growing. I ran into this because this logic needed to be fixed for ASAN (https://github.com/ruby/ruby/pull/11921) However, I actually think we should delete `reserve_stack` entirely, which is the point of this issue. Myself and @rianmcguire had a look at this today and we believe that the original problem was in fact a symptom of a kernel bug. The kernel bug (or at least, what we _think_ was the relevant bug) was fixed in 2017 (https://github.com/torvalds/linux/commit/c204d21f2232d875e36b8774c36ffd027d…) On my machine today, under ruby 3.3.2 (2024-05-30 revision e5a195edf6) and kernel 6.10.12-200.fc40.x86_64 I can no longer reproduce the problem demonstrated by the repro script (https://gist.github.com/csfrancis/46e360d401609275246c) ``` kjtsanaktsidis@kjtsanaktsidis-laptop ~ % ./repro.rb new minimum diff: 140730206072832 (2) new minimum diff: 140725853872128 (4) new minimum diff: 140723732975616 (10) new minimum diff: 140719631585280 (14) new minimum diff: 140719552581632 (69) new minimum diff: 140719410409472 (159) new minimum diff: 140719327940608 (1191) new minimum diff: 140719326601216 (3111) new minimum diff: 140719312199680 (6098) ``` Performing this kind of stack reservation actually causes _other_ problems - if RLIMIT_STACK is set to a high value, performing the eager mapping like this can fail for lack of real memory ``` kjtsanaktsidis@kjtsanaktsidis-laptop ~ % ulimit -s 1000000000 kjtsanaktsidis@kjtsanaktsidis-laptop ~ % ruby -e "puts 'hi'" zsh: segmentation fault (core dumped) ruby -e "puts 'hi'" ``` So, therefore, I believe the right thing to do is to just delete `reserve_stack`. Are there any objections to doing this? -- https://bugs.ruby-lang.org/

2 2

[ruby-core:112058] [Ruby master Bug#19383] Time.now.zone encoding for German display language in Windows is incorrect
by stringsn88keys (Thomas Powell) 21 Oct '24

21 Oct '24

Issue #19383 has been reported by stringsn88keys (Thomas Powell). ---------------------------------------- Bug #19383: Time.now.zone encoding for German display language in Windows is incorrect https://bugs.ruby-lang.org/issues/19383 * Author: stringsn88keys (Thomas Powell) * Status: Open * Priority: Normal * ruby -v: 3.1.3 * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- OS: Verified on Windows 10 and Windows Server 2022 and Ruby 2.7.7 through 3.1.3 Display language: Verified on German, but may impact other languages in which Time.now.zone returns characters that aren't [A-Za-z]. Time zone: CET (UTC +01:00) Amsterdam, Berlin, ... Time.now.zone # => "Mitteleuro\xE3ische Zeit" Time.now.zone.encoding # => #<Encoding:IBM437> puts Time.now.zone # => "Mitteleurop∑ische Zeit" (should be "Mitteleuropäische Zeit") Time.now.zone.encode(Encoding::UTF_8) # => "Mitteleurop∑ische Zeit" Doing a force_encoding on all encodings in Encoding.list reveals that ISO-8859-(1..16) and Windows-125(0,2,4,7) work to coerce the ä out of the time zone string: Time.now.zone.force_encoding(Encoding::WINDOWS_1252) # => "Mitteleuro\xE3ische Zeit" ... but ... Time.now.zone.force_encoding(Encoding::WINDOWS_1252).encode(Encoding::UTF_8) #=> "Mitteleuropäische Zeit" Related issue: This improper encoding/rendering caused Ohai's JSON output to be unparseable. Workaround was forcing to Windows-1252. https://github.com/chef/ohai/pull/1781 -- https://bugs.ruby-lang.org/

5 10

[ruby-core:117957] [Ruby master Bug#20500] Non-system directories are not searched when checking for jemalloc headers and libs, and building `enc`
by lish82 (Hiroki Katagiri) 21 Oct '24

21 Oct '24

Issue #20500 has been reported by lish82 (Hiroki Katagiri). ---------------------------------------- Bug #20500: Non-system directories are not searched when checking for jemalloc headers and libs, and building `enc` https://bugs.ruby-lang.org/issues/20500 * Author: lish82 (Hiroki Katagiri) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- I found a problem similar to #20494 on jemalloc It's similar to #20494, but it has a few more issues Problems: - Non-system directories (specified by `--with-opt-dir`) are not searched when checking jemalloc headers (almost the same as #20494) - Non-system directories are not searched when checking jemalloc libs - Non-system directories are not searched when building the `enc` directory sources I have confirmed that this problem can be fixed by applying the following patch: ```patch diff --git a/configure.ac b/configure.ac index 169662c..b2dce70 100644 --- a/configure.ac +++ b/configure.ac @@ -1392,6 +1392,8 @@ AC_ARG_WITH([jemalloc], [with_jemalloc=$withval], [with_jemalloc=no]) AS_IF([test "x$with_jemalloc" != xno],[ # find jemalloc header first + save_CPPFLAGS="${CPPFLAGS}" + CPPFLAGS="${INCFLAGS} ${CPPFLAGS}" malloc_header= AC_CHECK_HEADER(jemalloc/jemalloc.h, [malloc_header=jemalloc/jemalloc.h], [ AC_CHECK_HEADER(jemalloc.h, [malloc_header=jemalloc.h]) @@ -1423,6 +1425,8 @@ AS_IF([test "x$with_jemalloc" != xno],[ done done ]) + CPPFLAGS="${save_CPPFLAGS}" + unset save_CPPFLAGS with_jemalloc=${rb_cv_jemalloc_library} AS_CASE(["$with_jemalloc"], [no], diff --git a/enc/Makefile.in b/enc/Makefile.in index 6920bc9..ce93fdd 100644 --- a/enc/Makefile.in +++ b/enc/Makefile.in @@ -52,7 +52,7 @@ optflags = @optflags@ debugflags = @debugflags@ warnflags = @warnflags@ CCDLFLAGS = @CCDLFLAGS@ -INCFLAGS = -I. -I$(arch_hdrdir) -I$(hdrdir) -I$(top_srcdir) +INCFLAGS = -I. -I$(arch_hdrdir) -I$(hdrdir) -I$(top_srcdir) @incflags@ DEFS = @DEFS@ CPPFLAGS = @CPPFLAGS@ -DONIG_ENC_REGISTER=rb_enc_register LDFLAGS = @LDFLAGS@ ``` --- On a side note, I think there might be other parts affected by the changes bellow: https://github.com/ruby/ruby/pull/8449 I thought it might be a good idea to review everything thoroughly -- https://bugs.ruby-lang.org/

5 11

[ruby-core:117003] [Ruby master Bug#20314] Simultaneous Timeout expires may raise an exception after the block
by mame (Yusuke Endoh) 21 Oct '24

21 Oct '24

Issue #20314 has been reported by mame (Yusuke Endoh). ---------------------------------------- Bug #20314: Simultaneous Timeout expires may raise an exception after the block https://bugs.ruby-lang.org/issues/20314 * Author: mame (Yusuke Endoh) * Status: Open * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- Launchable reports `TestTimeout#test_nested_timeout` as a flaky test, and I reproduced it as follows. ```ruby require "timeout" class A < Exception end class B < Exception end begin Timeout.timeout(0.1, A) do Timeout.timeout(0.1, B) do nil while true end end rescue A, B p $! #=> #<A: execution expired> # Exception B is raised after the above call returns #=> test.rb:16:in `p': execution expired (B) p :end # not reach end ``` This is because the timer thread performs two consecutive `Thread#raise` to the target thread. I have discussed this with @ko1 and have come up with three solutions. ### Solution 1 When multiple nested Timeouts expire simultaneously, raise an exception for the outer-most Timeout and let the inner Timeouts expire without throwing an exception. In the above example, it would only raise A. The problem with this approach is that if you are rescuing A in the inner block, it may never ends: ```ruby Timeout.timeout(0.1, A) do Timeout.timeout(0.1, B) do begin sleep rescue A sleep # The exception A is caught. The inner Timeout is already expired, so the code (may) never end. end end end ``` Note that, if A and B did not occur at the same time, it would raise B. This is a race condition. ### Solution 2 When multiple nested Timeouts expire simultaneously, raise an exception for the inner-most Timeout and let the outer Timeouts wait until the inner-most Timeout returns. In the above example, it would raise either A or B, not both. The problem with this approach is that if you are rescuing B in the inner block, it never ends: ```ruby Timeout.timeout(0.1, A) do Timeout.timeout(0.1, B) do begin sleep rescue B sleep # The outer Timeout waits for the inner timeout, and the inner Timeout never return. So this code never ends. end end end ``` ### Solution 3 Make thread interrupt queue one length. If the target thread has already been `Thread#raise(A)`, the new `Thread#raise(B)` blocks until the target thread processes A. Since there will be no more simultaneous Thread#raise, there will be no more exceptions after the end of the block. The timeout timer thread should be changed in consideration that `Thread#raise` may block. -- https://bugs.ruby-lang.org/

3 4

[ruby-core:119551] [Ruby master Feature#16986] Anonymous Struct literal
by danh337 (Dan H) 21 Oct '24

21 Oct '24

Issue #16986 has been updated by danh337 (Dan H). Eregon (Benoit Daloze) wrote in #note-69: > That's pretty slow and inefficient as it creates a new Struct subclass for every Hash it's applied to. > The core idea of this proposal is to make it syntax and disallow **h to make sure that doesn't happen. Um ok so I'm voting against a new syntax. It's not necessary. A simple method like the one above is pretty good evidence. Whatever javascript is doing is not evidence for Ruby. When you say "slow and inefficient" you must be comparing to something else but it isn't clear what you're thinking. If you're comparing to a new notation which does not handle recursive nesting of Struct objects, then of course you're right. But then that notation would be even easier to obviate with a simple method. And then if a new syntax did handle nested Struct objects, then, um there are new Struct objects? Shrug. > This example could be a decent use case for OpenStruct, but that would need to be fixed perf-wise first, which is hard because of compatibility concerns: https://github.com/ruby/ostruct/issues/51 > Wrapping it in your own OpenStruct-like wrapper would likely be the best for now. Sorry I don't understand "best for now". This example is a way to AVOID OpenStruct. The nice things about Struct are (a) it catches typos, and (b) it's way faster than OpenStruct. I don't want smart Ruby folks like @Eregon to spend a lot of time on this. I just wanted to show that a new syntax is not really needed. ---------------------------------------- Feature #16986: Anonymous Struct literal https://bugs.ruby-lang.org/issues/16986#change-110165 * Author: ko1 (Koichi Sasada) * Status: Assigned * Assignee: matz (Yukihiro Matsumoto) ---------------------------------------- # Abstract How about introducing anonymous Struct literal such as `${a: 1, b: 2}`? It is almost the same as `Struct.new(:a, :b).new(1, 2)`. # Proposal ## Background In many cases, people use hash objects to represent a set of values such as `person = {name: "ko1", country: 'Japan'}` and access its values through `person[:name]` and so on. It is not easy to write (three characters `[:]`!), and it easily introduces misspelling (`person[:nama]` doesn't raise an error). If we make a `Struct` object by doing `Person = Struct.new(:name, :age)` and `person = Person.new('ko1', 'Japan')`, we can access its values through `person.name` naturally. However, it costs coding. And in some cases, we don't want to name the class (such as `Person`). Using `OpenStruct` (`person = OpenStruct.new(name: "ko1", country: "Japan")`), we can access it through `person.name`, but we can extend the fields unintentionally, and the performance is not good. Of course, we can define a class `Person` with attr_readers. But it takes several lines. To summarize the needs: * Easy to write * Doesn't require declaring the class * Accessible through `person.name` format * Limited fields * Better performance ## Idea Introduce new literal syntax for an anonymous Struct such as: `${ a: 1, b: 2 }`. Similar to Hash syntax (with labels), but with `$` prefix to distinguish. Anonymous structs which have the same member in the same order share their class. ```ruby s1 = ${a: 1, b: 2, c: 3} s2 = ${a: 1, b: 2, c: 3} assert s1 == s2 s3 = ${a: 1, c: 3, b: 2} s4 = ${d: 4} assert_equal false, s1 == s3 assert_equal false, s1 == s4 ``` ## Note Unlike Hash literal syntax, this proposal only allows `label: expr` notation. No `${**h}` syntax. This is because if we allow to splat a Hash, it can be a vulnerability by splatting outer-input Hash. Thanks to this spec, we can specify anonymous Struct classes at compile time. We don't need to find or create Struct classes at runtime. ## Implementatation https://github.com/ruby/ruby/pull/3259 # Discussion ## Notation Matz said he thought about `{|a: 1, b: 2 |}` syntax. ## Performance Surprisingly, Hash is fast and Struct is slow. ```ruby Benchmark.driver do |r| r.prelude <<~PRELUDE st = Struct.new(:a, :b).new(1, 2) hs = {a: 1, b: 2} class C attr_reader :a, :b def initialize() = (@a = 1; @b = 2) end ob = C.new PRELUDE r.report "ob.a" r.report "hs[:a]" r.report "st.a" end __END__ Warming up -------------------------------------- ob.a 38.100M i/s - 38.142M times in 1.001101s (26.25ns/i, 76clocks/i) hs[:a] 37.845M i/s - 38.037M times in 1.005051s (26.42ns/i, 76clocks/i) st.a 33.348M i/s - 33.612M times in 1.007904s (29.99ns/i, 87clocks/i) Calculating ------------------------------------- ob.a 87.917M i/s - 114.300M times in 1.300085s (11.37ns/i, 33clocks/i) hs[:a] 85.504M i/s - 113.536M times in 1.327850s (11.70ns/i, 33clocks/i) st.a 61.337M i/s - 100.045M times in 1.631064s (16.30ns/i, 47clocks/i) Comparison: ob.a: 87917391.4 i/s hs[:a]: 85503703.6 i/s - 1.03x slower st.a: 61337463.3 i/s - 1.43x slower ``` I believe we can speed up `Struct` similarly to ivar accesses, so we can improve the performance. BTW, OpenStruct (os.a) is slow. ``` Comparison: hs[:a]: 92835317.7 i/s ob.a: 85865849.5 i/s - 1.08x slower st.a: 53480417.5 i/s - 1.74x slower os.a: 12541267.7 i/s - 7.40x slower ``` For memory consumption, `Struct` is more lightweight because we don't need to keep the key names. ## Naming If we name an anonymous class, literals with the same members share the name. ```ruby s1 = ${a:1} s2 = ${a:2} p [s1, s2] #=> [#<struct a=1>, #<struct a=2>] A = s1.class p [s1, s2] #=> [#<struct A a=1>, #<struct A a=2>] ``` Maybe that is not a good behavior. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:119548] [Ruby master Feature#18336] How to deal with Trojan Source vulnerability
by wilburlo (Daniel Lo) 21 Oct '24

21 Oct '24

Issue #18336 has been updated by wilburlo (Daniel Lo). Please consider the issue of ASCII smuggling as a potential aspect of this problem. While I don’t currently see how ASCII smuggling could be used to affect Ruby, I do believe it would be worthwhile to explore if the command "ruby -c" should implement checks to detect bi-directional characters or ASCII smuggling. For more information, you may find this resource helpful: * https://embracethered.com/blog/posts/2024/hiding-and-finding-text-with-unic… ---------------------------------------- Feature #18336: How to deal with Trojan Source vulnerability https://bugs.ruby-lang.org/issues/18336#change-110164 * Author: duerst (Martin Dürst) * Status: Feedback ---------------------------------------- The "Torjan Source" vulnerability recently has caught some attention. The vulnerability involves using certain combinations of Unicode characters to let source code look like it is correct (and therefore pass code review,...) but actually do something else than intended. For background, please see discussion on KrebsonSecurity (https://krebsonsecurity.com/2021/11/trojan-source-bug-threatens-the-securit…) and the Web site (https://www.trojansource.codes/) and original paper (https://www.trojansource.codes/trojan-source.pdf) I contacted the Ruby security list, which was already aware of the issue, and we agreed to discuss this here because the vulnerability is already public. The paper focuses on the use of [A] Directional Formatting Characters (*1) in string constants, comments, and similar constructs to change the visual appearance of code outside these constructs. There are related vulnerabilities, namely the use of [B] non-spacing (and therefore mostly invisible) characters e.g. in variable names, and the use of [C] mixed-script identifiers, which also lets some variable names look identical even if they are not. Some languages, such as Rust, have addressed [A] (see https://blog.rust-lang.org/2021/11/01/cve-2021-42574.html) by requiring escapes to be used for the relevant characters in source. On the other hand, people such as Russ Cox think compilers are the wrong place to address the issue; it should be addressed in editors and similar tools (see https://research.swtch.com/trojan) Github now warns about The question is what Ruby should do, if anything. Addressing [A] similar to how Rust does it can be done relatively easily. If that's done, I'd prefer to only reject incomplete Bidi control sequences, which is a bit more complicated. In particular, string interpolation needs a very careful analysis. For [B], I'll open a separate issue. For [C], we have all data about scripts, but the way it's currently structured makes finding out which character a script belongs to quite inefficient. (*1) "Directional Formatting Character" is the official Unicode term (see https://www.unicode.org/reports/tr9/#Directional_Formatting_Characters) The terms "Bidi/Bidirectional control" or "Bidi/Bidirectional control character" are also used. Overall, there are 9 such characters. Unfortunately, both the paper and KrebsonSecurity use the term "Bidi Override", which is highly misleading. The term “Bidi Override” is reserved for two characters only: LRO, U+202D, Left-to-Right Override, and RLO, U+202E, Right-to-Left Override (see Table 1 in the paper). It is also used for the phenomenon associated with these two characters, a “hard” override (i.e. affecting all characters including e.g. the Latin alphabet), and mechanisms in other technology that achieve the same (e.g. the HTML bdo element (https://html.spec.whatwg.org/#the-bdo-element) or the ‘bidi-override’ value of the unicode-bidi property in CSS (https://www.w3.org/TR/CSS2/visuren.html#propdef-unicode-bidi)) -- https://bugs.ruby-lang.org/

1 0

[ruby-core:119352] [Ruby master Bug#20771] make-snapshot is broken with ubuntu-latest
by hsbt (Hiroshi SHIBATA) 21 Oct '24

21 Oct '24

Issue #20771 has been reported by hsbt (Hiroshi SHIBATA). ---------------------------------------- Bug #20771: make-snapshot is broken with ubuntu-latest https://bugs.ruby-lang.org/issues/20771 * Author: hsbt (Hiroshi SHIBATA) * Status: Closed * Assignee: hsbt (Hiroshi SHIBATA) * Backport: 3.1: REQUIRED, 3.2: REQUIRED, 3.3: REQUIRED ---------------------------------------- `make-snapshot` is broken now with the following failure. ``` Command Line Error: Unknown switch: -l creating zip archive... /home/runner/work/actions/actions/pkg/snapshot-ruby_3_3.zip 0.003 failed ``` I opened PRs to stable branches: * https://github.com/ruby/ruby/pull/11729 * https://github.com/ruby/ruby/pull/11730 * https://github.com/ruby/ruby/pull/11731 -- https://bugs.ruby-lang.org/

2 1

[ruby-core:119537] [Ruby master Bug#20802] It is possible to set the encoding of an IO instance to one that requires binmode when binmode is not set
by javanthropus (Jeremy Bopp) 21 Oct '24

21 Oct '24

Issue #20802 has been reported by javanthropus (Jeremy Bopp). ---------------------------------------- Bug #20802: It is possible to set the encoding of an IO instance to one that requires binmode when binmode is not set https://bugs.ruby-lang.org/issues/20802 * Author: javanthropus (Jeremy Bopp) * Status: Open * ruby -v: ruby 3.3.4 (2024-07-09 revision be1089c8ec) [x86_64-linux] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- I don't know what problems this may cause, but there are guards against setting the encoding of an IO instance to one such as UTF-16LE when the IO instance is not set to binary mode. Here is a method to bypass those guards: ```ruby f1 = File.open('/dev/null') f2 = File.open('/dev/null') f1.binmode f1.set_encoding('utf-16le') f2.set_encoding('utf-8') f1.reopen(f2) f1.binmode? # <= false f1.external_encoding # <= Encoding::UTF_16LE ``` -- https://bugs.ruby-lang.org/

1 0

[ruby-core:119536] [Ruby master Bug#20801] Handling non-ascii drive letter on Windows
by mame (Yusuke Endoh) 21 Oct '24

21 Oct '24

Issue #20801 has been reported by mame (Yusuke Endoh). ---------------------------------------- Bug #20801: Handling non-ascii drive letter on Windows https://bugs.ruby-lang.org/issues/20801 * Author: mame (Yusuke Endoh) * Status: Feedback * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- On Windows, non-ascii drive letters are allowed for virtual drives. Some Ruby methods do not seem to handle this well. ``` subst ä: C:\tmp\ ``` ```ruby File.absolute_path?("ä:/") #=> expected: true, actual: false File.write("ä:/foo.txt", "foo") Dir.glob("c:/tmp/*.txt") #=> ["c:/tmp/foo.txt"] Dir.glob("ä:/*.txt") #=> expected: ["ä:/foo.txt"], actual: [] Dir.entries("ä:/") #=> [".", "..", "foo.txt"] ``` It would be nice if someone more familiar with Windows could do a more comprehensive investigation and write a patch. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:119533] [Ruby master Bug#20799] Bug in forwarding to struct methods
by tenderlovemaking (Aaron Patterson) 21 Oct '24

21 Oct '24

Issue #20799 has been reported by tenderlovemaking (Aaron Patterson). ---------------------------------------- Bug #20799: Bug in forwarding to struct methods https://bugs.ruby-lang.org/issues/20799 * Author: tenderlovemaking (Aaron Patterson) * Status: Closed * Assignee: tenderlovemaking (Aaron Patterson) * ruby -v: ruby 3.4.0dev (2024-10-15T18:34:24Z master f45eb3dcb9) +PRISM [arm64-darwin24] * Backport: 3.1: DONTNEED, 3.2: DONTNEED, 3.3: DONTNEED ---------------------------------------- This program crashes: ```ruby Thing = Struct.new(:value) Obj = Thing.new("ok") def delegate(...) Obj.value(...) end def no_args delegate end def splat_args(*args) delegate(*args) end no_args splat_args ``` It's crashing because we're forwarding to a struct method without taking in to account the forwarding IC. I have a patch prepared, but I am filing this ticket to track it. -- https://bugs.ruby-lang.org/

1 0