August 2024 - ruby-core - ml.ruby-lang.org

[ruby-core:118782] [Ruby master Bug#20663] Reading from IO does not recover gracefully from bad data pushed via IO#ungetc
by javanthropus (Jeremy Bopp) 09 Aug '24

09 Aug '24

Issue #20663 has been reported by javanthropus (Jeremy Bopp). ---------------------------------------- Bug #20663: Reading from IO does not recover gracefully from bad data pushed via IO#ungetc https://bugs.ruby-lang.org/issues/20663 * Author: javanthropus (Jeremy Bopp) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- If bytes that result in at least 2 invalid characters for the internal encoding of an IO object are pushed into the internal buffer with IO#getc, reading from the stream returns invalid characters composed of both bytes from the internal buffer and the converted bytes from the stream even if the next character in the stream itself is completely valid. ``` ruby char_bytes = Tempfile.open(encoding: 'utf-8:utf-16le') do |f| f.write("🍣") f.rewind f.ungetc("🍣".encode('utf-16le').b[0..-2]) f.each_char.map(&:bytes) end puts char_bytes.inspect ``` The above outputs: ``` [[60, 216], [99, 60], [216, 99], [223]] ``` I expect it to output: ``` [[60, 216], [99], [60, 216, 99, 223]] ``` In other words, I expect it to first completely drain the internal character buffer returning as many characters as necessary (invalid or otherwise) before reading from the stream and converting and returning the next character. Interestingly, if there are only bytes sufficient for 1 invalid character in the internal buffer, it behaves that way: ``` ruby char_bytes = Tempfile.open(encoding: 'utf-8:utf-16le') do |f| f.write("🍣") f.rewind f.ungetc("🍣".encode('utf-16le').b[0..-3]) # <- Note the -3 here vs. the -2 earlier f.each_char.map(&:bytes) end puts char_bytes.inspect ``` This outputs: ``` [[60, 216], [60, 216, 99, 223]] ``` The first character is invalid, but returning it first clears the buffer. Then the next character is read, converted, and returned in full. -- https://bugs.ruby-lang.org/

1 1

[ruby-core:118588] [Ruby master Bug#20634] ruby uses 1/2 (7 out of 16) cores for 16 ractors with RUBY_MAX_CPU=16 environment variable set
by skorobogatydmitry (Dmitry Skorobogaty) 09 Aug '24

09 Aug '24

Issue #20634 has been reported by skorobogatydmitry (Dmitry Skorobogaty). ---------------------------------------- Bug #20634: ruby uses 1/2 (7 out of 16) cores for 16 ractors with RUBY_MAX_CPU=16 environment variable set https://bugs.ruby-lang.org/issues/20634 * Author: skorobogatydmitry (Dmitry Skorobogaty) * Status: Open * ruby -v: ruby 3.3.1 (2024-04-23 revision c56cd86388) [x86_64-linux] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- It's a cont for #20618, feel free to close this one and reopen the original bug. According to `top`, the below code uses 7/16 cores (700%) of my AMD Ryzen 7 5800HS: #!/usr/bin/env ruby 16.times { Ractor.new { 10_000.downto(1) { |i| 100_000.downto(1) { |j| i * j } } } } sleep 30 ``` Command I ran is `RUBY_MAX_CPU=16 ruby test.rb`. I also tried to set `RUBY_MN_THREADS=1` as an addition and the script starts to occupy 8 cores instead of 7 (possibly, due to N:M algo allocating an extra core for the blocked main Ractor). The same code occupies all 16 cores on ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [x86_64-linux] The problem reproduces on ruby 3.3.3 (2024-06-12 revision f1c7b6f435) [x86_64-linux] (ruby from the latest docker image). -- https://bugs.ruby-lang.org/

1 2

[ruby-core:118789] [Ruby master Bug#20665] Swig testing of Ruby with MinGW-w64 UCRT
by ErezGeva2＠gmail.com (Erez Geva) 07 Aug '24

07 Aug '24

Issue #20665 has been reported by ErezGeva2(a)gmail.com (Erez Geva). ---------------------------------------- Bug #20665: Swig testing of Ruby with MinGW-w64 UCRT https://bugs.ruby-lang.org/issues/20665 * Author: ErezGeva2(a)gmail.com (Erez Geva) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- Hi, I try to add SWIG project testing with already installed Ruby on [GitHub Windows2022](https://github.com/actions/runner-images/blob/main/images/wind…. I manage to test with Ruby version 3.0.7 and installed mingw-w64-x86_64-ruby version 3.1.5. Both are build with MinGW-w64. See: https://github.com/swig/swig/pull/2981 Now I want to use the more newer Ruby. GitHub Windows2022 provides 2 Ruby version 3.1.6 and 3.2.5 which are build with MinGW-w64 UCRT. I try and use mingw-w64-ucrt-x86_64-gcc with more mingw-w64-ucrt-x86_64-XXX packages. My testing can be found here: https://github.com/erezgeva/swig/blob/win_ruby2/.github/workflows/windows.y… The test fails: https://github.com/erezgeva/swig/actions/runs/10237377988/job/28320593017 Can you advice what is wrong here? Erez -- https://bugs.ruby-lang.org/

2 4

[ruby-core:118520] [Ruby master Feature#20621] Check libruby.so hardening by annocheck
by vo.x (Vit Ondruch) 07 Aug '24

07 Aug '24

Issue #20621 has been reported by vo.x (Vit Ondruch). ---------------------------------------- Feature #20621: Check libruby.so hardening by annocheck https://bugs.ruby-lang.org/issues/20621 * Author: vo.x (Vit Ondruch) * Status: Open ---------------------------------------- As part of #18061, there was implemented annocheck test case. However, the test covers just `ruby` (which is just thin shell), while leaves out `libruby.so` (which contains most of the Ruby code). This PR tries to improve that: https://github.com/ruby/ruby/pull/11123 BTW the question is if all *.so files or even all *.o files should be covered. -- https://bugs.ruby-lang.org/

2 2

[ruby-core:118802] [Ruby master Feature#17166] net/http not supporting unix domain sockets
by kddnewton (Kevin Newton) 06 Aug '24

06 Aug '24

Issue #17166 has been updated by kddnewton (Kevin Newton). Then build it. If it’s as widely used as you say, the PR is sure to be accepted. ---------------------------------------- Feature #17166: net/http not supporting unix domain sockets https://bugs.ruby-lang.org/issues/17166#change-109358 * Author: hadmut (Hadmut Danisch) * Status: Third Party's Issue ---------------------------------------- Hi, meanwhile it's common and state of the art to offer rest apis not just over tcp/ip, but over unix domain sockets as well for machine-internal use and advanced security. Unfortunately net/http does not support unix domain sockets. Although there is a workaround with the gem net_http_unix / NetX::HTTPUnix, this is rather useless, since most programs, libs, gems (e.g. rest-client) are based on net/http. However, there's some security consideration. When evaluating e.g. HTML and accessing URLs, it could be harmful or leak information, if an URL point to some unix domain path could reveal information or allow to trigger something, e.g. tell through error messages whether a file exists or let someone unintentionally install a packet through ubuntu's snap mechanism (which is controlled through a unix domain socket with rest api). It should, however, be possible to use unix domain sockets (without workaround, third party gem or low level code). regards -- https://bugs.ruby-lang.org/

1 0

[ruby-core:118801] [Ruby master Feature#17166] net/http not supporting unix domain sockets
by hadmut (Hadmut Danisch) 06 Aug '24

06 Aug '24

Issue #17166 has been updated by hadmut (Hadmut Danisch). That's simply wrong. Several widely used apps offer their REST API over unix domain sockets for security reasons, e.g. LXD, snapd, docker, podman, which are just four of the most commonly used applications in the Linux world. How can you call it a "very fringe use case" and "extremely rare"? Missed the last 10 years of software development? ---------------------------------------- Feature #17166: net/http not supporting unix domain sockets https://bugs.ruby-lang.org/issues/17166#change-109357 * Author: hadmut (Hadmut Danisch) * Status: Third Party's Issue ---------------------------------------- Hi, meanwhile it's common and state of the art to offer rest apis not just over tcp/ip, but over unix domain sockets as well for machine-internal use and advanced security. Unfortunately net/http does not support unix domain sockets. Although there is a workaround with the gem net_http_unix / NetX::HTTPUnix, this is rather useless, since most programs, libs, gems (e.g. rest-client) are based on net/http. However, there's some security consideration. When evaluating e.g. HTML and accessing URLs, it could be harmful or leak information, if an URL point to some unix domain path could reveal information or allow to trigger something, e.g. tell through error messages whether a file exists or let someone unintentionally install a packet through ubuntu's snap mechanism (which is controlled through a unix domain socket with rest api). It should, however, be possible to use unix domain sockets (without workaround, third party gem or low level code). regards -- https://bugs.ruby-lang.org/

1 0

[ruby-core:118800] [Ruby master Feature#17166] net/http not supporting unix domain sockets
by byroot (Jean Boussier) 06 Aug '24

06 Aug '24

Issue #17166 has been updated by byroot (Jean Boussier). Status changed from Open to Third Party's Issue No it means it's a very fringe use case and nobody with that use case (including you) bothered to work on it. It's extremely rare to use HTTP over unix sockets, and even very popular HTTP libraries for very popular languages like Python's `requests` [don't support it and require monkey patching](https://github.com/msabramo/requests-unixsocket) and nobody calls Python dead. So if you really need this you are welcome to open a pull request at https://github.com/ruby/net-http ---------------------------------------- Feature #17166: net/http not supporting unix domain sockets https://bugs.ruby-lang.org/issues/17166#change-109356 * Author: hadmut (Hadmut Danisch) * Status: Third Party's Issue ---------------------------------------- Hi, meanwhile it's common and state of the art to offer rest apis not just over tcp/ip, but over unix domain sockets as well for machine-internal use and advanced security. Unfortunately net/http does not support unix domain sockets. Although there is a workaround with the gem net_http_unix / NetX::HTTPUnix, this is rather useless, since most programs, libs, gems (e.g. rest-client) are based on net/http. However, there's some security consideration. When evaluating e.g. HTML and accessing URLs, it could be harmful or leak information, if an URL point to some unix domain path could reveal information or allow to trigger something, e.g. tell through error messages whether a file exists or let someone unintentionally install a packet through ubuntu's snap mechanism (which is controlled through a unix domain socket with rest api). It should, however, be possible to use unix domain sockets (without workaround, third party gem or low level code). regards -- https://bugs.ruby-lang.org/

1 0

[ruby-core:118794] [Ruby master Feature#17166] net/http not supporting unix domain sockets
by hadmut (Hadmut Danisch) 05 Aug '24

05 Aug '24

Issue #17166 has been updated by hadmut (Hadmut Danisch). Now this is almost four years old without any reaction at all. Has anyone ever taken any notice of this problem, which affects a lot of people which are those REST API frameworks? Or is ruby dead, and it's time to find a new language? ---------------------------------------- Feature #17166: net/http not supporting unix domain sockets https://bugs.ruby-lang.org/issues/17166#change-109349 * Author: hadmut (Hadmut Danisch) * Status: Open ---------------------------------------- Hi, meanwhile it's common and state of the art to offer rest apis not just over tcp/ip, but over unix domain sockets as well for machine-internal use and advanced security. Unfortunately net/http does not support unix domain sockets. Although there is a workaround with the gem net_http_unix / NetX::HTTPUnix, this is rather useless, since most programs, libs, gems (e.g. rest-client) are based on net/http. However, there's some security consideration. When evaluating e.g. HTML and accessing URLs, it could be harmful or leak information, if an URL point to some unix domain path could reveal information or allow to trigger something, e.g. tell through error messages whether a file exists or let someone unintentionally install a packet through ubuntu's snap mechanism (which is controlled through a unix domain socket with rest api). It should, however, be possible to use unix domain sockets (without workaround, third party gem or low level code). regards -- https://bugs.ruby-lang.org/

1 0

[ruby-core:118621] [Ruby master Misc#20639] Request to host C API docs on docs.ruby-lang.org
by alanwu (Alan Wu) 02 Aug '24

02 Aug '24

Issue #20639 has been reported by alanwu (Alan Wu). ---------------------------------------- Misc #20639: Request to host C API docs on docs.ruby-lang.org https://bugs.ruby-lang.org/issues/20639 * Author: alanwu (Alan Wu) * Status: Open ---------------------------------------- Having a first-party documentation site benefits readers and helps contributors improve the contents. The Ruby standard library API benefits from being hosted on <https://docs.ruby-lang.org> and it would be nice if the C API documentation produced by Doxygen can be hosted on the same site as well. I'm not familiar with the logistics of hosting the documentation site, and traffic is not free. But I think this is worth considering for the benefit of Ruby extension authors and maintainers. At the moment, <https://github.com/ruby/actions> generates the documentation with Doxygen but it's not hosted anywhere as far as I can tell. Searching for "Ruby C API documentation" on Google lead to guides about the API and no results that provide a comprehensive up-to-date reference. -- https://bugs.ruby-lang.org/

3 5

[ruby-core:118778] [Ruby master Feature#14602] Version of dig that raises error if a key is not present
by jordan-brough (Jordan Brough) 02 Aug '24

02 Aug '24

Issue #14602 has been updated by jordan-brough (Jordan Brough). > Matz was interested in how dig_fetch or fetch_dig would sound to English native speakers. As a native english speaker, "dig_fetch" sounds natural and makes sense to me. ---------------------------------------- Feature #14602: Version of dig that raises error if a key is not present https://bugs.ruby-lang.org/issues/14602#change-109331 * Author: amcaplan (Ariel Caplan) * Status: Open ---------------------------------------- Currently, if I have a hash like this: ~~~ ruby { :name => { :first => "Ariel", :last => "Caplan" } } ~~~ and I want to navigate confidently and raise a KeyError if something is missing, I can do: ~~~ ruby hash.fetch(:name).fetch(:first) ~~~ Unfortunately, the length of the name, combined with the need to repeat the method name every time, means most programmers are more likely to do this: ~~~ ruby hash[:name][:first] ~~~ which leads to many unexpected errors. The Hash#dig method made it easy to access methods safely from a nested hash; I'd like to have something similar for access without error protection, and I'd think the most natural name would be Hash#dig!. It would work like this: ~~~ ruby hash = { :name => { :first => "Ariel", :last => "Caplan" } } hash.dig!(:name, :first) # => Ariel hash.dig!(:name, :middle) # raises KeyError (key not found: :middle) hash.dig!(:name, :first, :foo) # raises TypeError (String does not have #dig! method) ~~~ -- https://bugs.ruby-lang.org/

1 0