- ruby-core - ml.ruby-lang.org

[ruby-core:113643] [Ruby master Misc#19692] Net::HTTP Performance Workstream
by baweaver (Brandon Weaver) 24 May '23

24 May '23

Issue #19692 has been reported by baweaver (Brandon Weaver). ---------------------------------------- Misc #19692: Net::HTTP Performance Workstream https://bugs.ruby-lang.org/issues/19692 * Author: baweaver (Brandon Weaver) * Status: Open * Priority: Normal ---------------------------------------- While working on identifying causes behind Capybara test slowness I had noticed some memory profiles pointed at a few slow code paths in `Net::HTTP`, one of which I've already opened a PR for here: https://github.com/ruby/net-http/pull/140 The reason I'm opening this ticket is that I had found some additional areas which warrant investigation for some decent performance gains in frequently run sections of code in `Net::HTTP` and to ask if someone would be willing to work with me on opening likely several PRs against the code base in the near future. I will hold myself to the standard of qualifying these changes with performance data on which parts are frequently run and are slow, but would appreciate working with someone from core through this. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:113623] [Ruby master Feature#19690] Add terminal colorizing gem to stdlib
by ccmywish (Aoran Zeng) 24 May '23

24 May '23

Issue #19690 has been reported by ccmywish (Aoran Zeng). ---------------------------------------- Feature #19690: Add terminal colorizing gem to stdlib https://bugs.ruby-lang.org/issues/19690 * Author: ccmywish (Aoran Zeng) * Status: Open * Priority: Normal ---------------------------------------- Recently the new version of `tracer` gem caught my eye. But I've noticed an anti-DRY pattern. I've noticed that many libs, CLI tools implement their own logic of color support on the terminal. These libs/CLI tools don't explicitly require a gem from RubyGems.org may be because of these reasons: 1. It's very easy to directly monkey patch `String` class to support, such as `bold`, `red`, `blue` 2. The lib is very independent, the author doesn't want any dependency. See: https://github.com/ruby/tracer/blob/master/lib/tracer/color.rb I hope the core team consider adding `official` support for terminal colorizing, because most Ruby developers live on the terminal. If we've added it, all gems/CLI tools developers can easily add simple styles. For example, adding color to `gem` and `bundler` command is attractive. Next door community's `npm` seem much cooler by just using simple colors. Could we coordinate with the gem [rainbow](https://rubygems.org/gems/rainbow)'s author to make it one of the stdlib or bundled gem like `minitest`? -- https://bugs.ruby-lang.org/

4 4

[ruby-core:113630] [Ruby master Misc#19122] Use MADV_DONTNEED instead of MADV_FREE when freeing a Fiber's stack
by ioquatix (Samuel Williams) 24 May '23

24 May '23

Issue #19122 has been updated by ioquatix (Samuel Williams). https://github.com/ruby/ruby/pull/7855 should enable you to test different advice. Just bit shift it to the left to enable any `MADV_...` value to be specified. ---------------------------------------- Misc #19122: Use MADV_DONTNEED instead of MADV_FREE when freeing a Fiber's stack https://bugs.ruby-lang.org/issues/19122#change-103271 * Author: smcgivern (Sean McGivern) * Status: Open * Priority: Normal * Assignee: ioquatix (Samuel Williams) ---------------------------------------- I'd like to propose that Ruby stops using MADV_FREE when freeing a Fiber's stack, and switches to using MADV_DONTNEED even when MADV_FREE is supported. MADV_FREE is used in one place in the Ruby codebase, when freeing the stack of a freed Fiber: https://git.ruby-lang.org/ruby.git/tree/cont.c#n683 The comment for `fiber_pool_stack_free` says: ```c // We advise the operating system that the stack memory pages are no longer being used. // This introduce some performance overhead but allows system to relaim memory when there is pressure. ``` Where possible (i.e. on Linux 4.5 and later), `fiber_pool_stack_free` uses `MADV_FREE` over `MADV_DONTNEED`. This has the side effect that memory statistics such as RSS will not reduce until and unless the OS actually reclaims that memory. If that doesn't happen, then the reported memory usage via RSS will be much higher than the 'real' memory usage. If this was pervasive throughtout the Ruby codebase then that would be one thing, but currently this is just for Fiber. This means that: 1. A program that doesn't use Fiber will have somewhat reliable RSS statistics on recent Linux. 2. A program that heavily uses Fiber (such as something using Async::HTTP) will see an inflated RSS statistic. Go made a similar change to the one I'm proposing here for similar reasons: https://github.com/golang/go/issues/42330 > While `MADV_FREE` is somewhat faster than `MADV_DONTNEED`, it doesn't affect many of the statistics that `MADV_DONTNEED` does until the memory is actually reclaimed. This generally leads to poor user experience, like confusing stats in `top` and other monitoring tools; and bad integration with management systems that respond to memory usage. > [...] > I propose we change the default to prefer `MADV_DONTNEED` over `MADV_FREE`, to favor user-friendliness and minimal surprise over performance. I think it's become clear that Linux's implementation of `MADV_FREE` ultimately doesn't meet our needs. As an aside, MADV_FREE was not used in Ruby 3.1 (https://bugs.ruby-lang.org/issues/19101) and I haven't found any bugs filed about this behaviour other than that one. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:113626] [Ruby master Bug#16265] Test (spec) failure using current MSYS2 tools, related to -fstack-protector and possibly D_FORTIFY_SOURCE=2
by ioquatix (Samuel Williams) 24 May '23

24 May '23

Issue #16265 has been updated by ioquatix (Samuel Williams). I'm not sure how the `ubf_io` can be used as part of the public extension API on Windows, as it's unable to interrupt non-Ruby IO methods, i.e. it only works when using `rb_w32_read` and `rb_w32_write`. Maybe the test should use actual IO objects, and invoke the equivalent `#read` and `#write` methods rather than depending on the native wrappers. ---------------------------------------- Bug #16265: Test (spec) failure using current MSYS2 tools, related to -fstack-protector and possibly D_FORTIFY_SOURCE=2 https://bugs.ruby-lang.org/issues/16265#change-103267 * Author: MSP-Greg (Greg L) * Status: Closed * Priority: Normal * Assignee: ioquatix (Samuel Williams) * ruby -v: ruby 2.7.0dev (2019-10-19T08:19:27Z master 1d435bd51a) [x64-mingw32] * Backport: 2.5: UNKNOWN, 2.6: UNKNOWN ---------------------------------------- GH PR https://github.com/ruby/ruby/pull/2573 was a work-around that forced using some non-current MSYS2 packages libraries for building. These libraries are only available on Appveyor because their images are out-of-date, a new MSYS2 install would not have them. Using current MSYS2 libraries requires adding '-fstack-protector' to LDFLAGS. I also experimented with adding 'D_FORTIFY_SOURCE=2' to CFLAGS. Ruby build/compiles with those flags added. test-all passes. But spec tests always lock up when running: ``` C-API Thread function rb_thread_call_without_gvl - runs a C function with the global lock unlocked - runs a C function with the global lock unlocked amd unlocks IO with the generic RUBY_UBF_IO ``` The tests are in spec/ruby/optional/capi/thread_spec.rb, as shown: https://github.com/ruby/ruby/blob/1d435bd51aeca75f6e437bc92de2271a716a1ced/… and the freeze occurs on line 137. I've run this once in ruby-loco, twice in my ruby fork, and several times locally. In all runs, test-all passed, and spec stopped at the test above. This uses gcc 9.2.0, and the problem would occur with anyone doing a 'fresh' install of MSYS2. Not being a c type, this is way beyond what I can fix. Happy to try patches, etc. Thanks, Greg -- https://bugs.ruby-lang.org/

1 0

[ruby-core:113624] [Ruby master Bug#16265] Test (spec) failure using current MSYS2 tools, related to -fstack-protector and possibly D_FORTIFY_SOURCE=2
by ioquatix (Samuel Williams) 24 May '23

24 May '23

Issue #16265 has been updated by ioquatix (Samuel Williams). Status changed from Open to Closed Merged in <https://github.com/ruby/ruby/commit/6d976eb5348098a346d82065621e37925acae8b8>. ---------------------------------------- Bug #16265: Test (spec) failure using current MSYS2 tools, related to -fstack-protector and possibly D_FORTIFY_SOURCE=2 https://bugs.ruby-lang.org/issues/16265#change-103265 * Author: MSP-Greg (Greg L) * Status: Closed * Priority: Normal * Assignee: ioquatix (Samuel Williams) * ruby -v: ruby 2.7.0dev (2019-10-19T08:19:27Z master 1d435bd51a) [x64-mingw32] * Backport: 2.5: UNKNOWN, 2.6: UNKNOWN ---------------------------------------- GH PR https://github.com/ruby/ruby/pull/2573 was a work-around that forced using some non-current MSYS2 packages libraries for building. These libraries are only available on Appveyor because their images are out-of-date, a new MSYS2 install would not have them. Using current MSYS2 libraries requires adding '-fstack-protector' to LDFLAGS. I also experimented with adding 'D_FORTIFY_SOURCE=2' to CFLAGS. Ruby build/compiles with those flags added. test-all passes. But spec tests always lock up when running: ``` C-API Thread function rb_thread_call_without_gvl - runs a C function with the global lock unlocked - runs a C function with the global lock unlocked amd unlocks IO with the generic RUBY_UBF_IO ``` The tests are in spec/ruby/optional/capi/thread_spec.rb, as shown: https://github.com/ruby/ruby/blob/1d435bd51aeca75f6e437bc92de2271a716a1ced/… and the freeze occurs on line 137. I've run this once in ruby-loco, twice in my ruby fork, and several times locally. In all runs, test-all passed, and spec stopped at the test above. This uses gcc 9.2.0, and the problem would occur with anyone doing a 'fresh' install of MSYS2. Not being a c type, this is way beyond what I can fix. Happy to try patches, etc. Thanks, Greg -- https://bugs.ruby-lang.org/

1 0

[ruby-core:113619] [Ruby master Bug#18738] IRB can't recognize heredoc after words
by ccmywish (Aoran Zeng) 24 May '23

24 May '23

Issue #18738 has been updated by ccmywish (Aoran Zeng). This has been fixed. Please close it. ---------------------------------------- Bug #18738: IRB can't recognize heredoc after words https://bugs.ruby-lang.org/issues/18738#change-103263 * Author: ccmywish (Aoran Zeng) * Status: Open * Priority: Normal * ruby -v: ruby 3.1.1p18 (2022-02-18 revision 53f5fc4236) [x86_64-linux] * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN ---------------------------------------- My irb_info ```ruby irb(main):001:0> irb_info => Ruby version: 3.1.1 IRB version: irb 1.4.1 (2021-12-25) InputMethod: ReidlineInputMethod with Reline 0.3.1 RUBY_PLATFORM: x86_64-linux LANG env: en_US.UTF-8 East Asian Ambiguous Width: 1 ``` See the code below please. ```ruby a, b = <<EOF, %w[ hello thank you ruby devs EOF world ] p a p b ``` This works well if you save it to a file, and run with `ruby xxx.rb`. The results are here: ```ruby "thank you\nruby devs\n" ["hello", "world"] ``` But when you type it to `irb`，the code will not end, and you will get: ```ruby ❯ irb irb(main):001:0] a, b = <<EOF, %w[ hello irb(main):002:0] thank you irb(main):003:0] ruby devs irb(main):004:0] EOF irb(main):005:0] world irb(main):006:0" ] irb(main):007:-" irb(main):008:0" irb(main):009:0" irb(main):010:0" ``` I found this issue when I read the mruby source code. in mruby, the token after the first line's `hello` should be `tHD_LITERAL_DELIM`. But in CRuby, there's no this token. I tried to dump CRuby's parser state, find that just after reading `<<EOF`，it will directly recognize the whole token 'thank you\nruby\devs'. So, I think this may not be the bug of Ripper, but how IRB called Ripper using its `ruby-lex` line by line. For your convenience, you can see the parser state. ``` Stack now 0 2 82 341 Entering state 580 Next token is token "string literal" (1.7-1.12: ) Shifting token "string literal" (1.7-1.12: ) Entering state 60 Reducing stack by rule 613 (line 4830): -> $$ = nterm string_contents (1.12-1.12: ) Stack now 0 2 82 341 580 60 Entering state 301 Reading a token: Next token is token "literal content" (1.12-1.12: "thank you\nruby devs\n") Shifting token "literal content" (1.12-1.12: "thank you\nruby devs\n") Entering state 507 Reducing stack by rule 619 (line 4926): $1 = token "literal content" (1.12-1.12: "thank you\nruby devs\n") -> $$ = nterm string_content (1.12-1.12: ) Stack now 0 2 82 341 580 60 301 Entering state 511 Reducing stack by rule 614 (line 4840): $1 = nterm string_contents (1.12-1.12: ) $2 = nterm string_content (1.12-1.12: ) -> $$ = nterm string_contents (1.12-1.12: ) Stack now 0 2 82 341 580 60 Entering state 301 Reading a token: lex_state: BEG -> END at line 7453 Next token is token "terminator" (1.12-1.12: ) Shifting token "terminator" (1.12-1.12: ) Entering state 512 Reducing stack by rule 596 (line 4693): $1 = token "string literal" (1.7-1.12: ) $2 = nterm string_contents (1.12-1.12: ) $3 = token "terminator" (1.12-1.12: ) -> $$ = nterm string1 (1.7-1.12: ) Stack now 0 2 82 341 580 Entering state 109 Reducing stack by rule 594 (line 4683): $1 = nterm string1 (1.7-1.12: ) -> $$ = nterm string (1.7-1.12: ) Stack now 0 2 82 341 580 Entering state 108 Reading a token: lex_state: END -> BEG|LABEL at line 9814 Next token is token ',' (1.12-1.13: ) ``` -- https://bugs.ruby-lang.org/

1 0

[ruby-core:113601] [Ruby master Bug#19687] Should a development version of the standard library be included in ruby/ruby?
by jaruga (Jun Aruga) 24 May '23

24 May '23

Issue #19687 has been reported by jaruga (Jun Aruga). ---------------------------------------- Bug #19687: Should a development version of the standard library be included in ruby/ruby? https://bugs.ruby-lang.org/issues/19687 * Author: jaruga (Jun Aruga) * Status: Open * Priority: Normal * ruby -v: ruby 3.3.0dev (2023-05-22T03:58:17Z master bd786e7896) [x86_64-linux] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- The `stringio` gem uses the value of the `STRINGIO_VERSION` as the gem version. https://github.com/ruby/ruby/blob/31ac8efca8ecb574e1e7b7c32cce54cb1b97f19a/… https://github.com/ruby/ruby/blob/31ac8efca8ecb574e1e7b7c32cce54cb1b97f19a/… ``` #define STRINGIO_VERSION "3.0.7" ``` It seems that that `stringio` on the current master branch in ruby/ruby sets the version 3.0.7 that doesn't exist in the RubyGems <https://rubygems.org/gems/stringio> by the commit <https://github.com/ruby/ruby/commit/67743d582317544d2c830b63d3423b04bad185a7>. And I see that this causes raising an error when running `bundle install` on ruby/openssl with Ruby on the master branch. I captured the `Gemeifile.lock` by `bundle lock`. And I see the `rdoc` depending on `psych` depending on `stringio`. The workaround was to add `gem "stringio", "< 3.0.7"` to the `Gemfile`. However, it's not convenient. Is it an expected behavior? Or should the ruby/ruby always merge the static version of the ruby/* (upstream standard libraries such as "stringio") rater than the development version? ``` $ which ruby ~/.local/ruby-bd786e7896/bin/ruby $ ruby -v ruby 3.3.0dev (2023-05-22T03:58:17Z master bd786e7896) [x86_64-linux] $ pwd /home/jaruga/git/ruby/openssl $ bundle install --standalone ... stringio-3.0.7 is built in to Ruby, and can't be cached because your Gemfile doesn't have any sources that contain it. ... Bundler::GemNotFound: Could not find stringio-3.0.7.gem for installation /home/jaruga/.local/ruby-bd786e7896/lib/ruby/3.3.0+0/bundler/source/rubygems.rb:157:in `install' /home/jaruga/.local/ruby-bd786e7896/lib/ruby/3.3.0+0/bundler/installer/gem_installer.rb:54:in `install' /home/jaruga/.local/ruby-bd786e7896/lib/ruby/3.3.0+0/bundler/installer/gem_installer.rb:16:in `install_from_spec' /home/jaruga/.local/ruby-bd786e7896/lib/ruby/3.3.0+0/bundler/installer/parallel_installer.rb:156:in `do_install' /home/jaruga/.local/ruby-bd786e7896/lib/ruby/3.3.0+0/bundler/installer/parallel_installer.rb:147:in `block in worker_pool' /home/jaruga/.local/ruby-bd786e7896/lib/ruby/3.3.0+0/bundler/worker.rb:62:in `apply_func' /home/jaruga/.local/ruby-bd786e7896/lib/ruby/3.3.0+0/bundler/worker.rb:57:in `block in process_queue' <internal:kernel>:187:in `loop' /home/jaruga/.local/ruby-bd786e7896/lib/ruby/3.3.0+0/bundler/worker.rb:54:in `process_queue' /home/jaruga/.local/ruby-bd786e7896/lib/ruby/3.3.0+0/bundler/worker.rb:90:in `block (2 levels) in create_threads' An error occurred while installing stringio (3.0.7), and Bundler cannot continue. ``` -- https://bugs.ruby-lang.org/

2 8

[ruby-core:113615] [Ruby master Bug#16265] Test (spec) failure using current MSYS2 tools, related to -fstack-protector and possibly D_FORTIFY_SOURCE=2
by ioquatix (Samuel Williams) 24 May '23

24 May '23

Issue #16265 has been updated by ioquatix (Samuel Williams). @shyouhei Do you think we should add something like: ``` #ifdef _WIN32 #include "ruby/win32.h" #endif ``` to `ruby.h`? ---------------------------------------- Bug #16265: Test (spec) failure using current MSYS2 tools, related to -fstack-protector and possibly D_FORTIFY_SOURCE=2 https://bugs.ruby-lang.org/issues/16265#change-103259 * Author: MSP-Greg (Greg L) * Status: Open * Priority: Normal * Assignee: ioquatix (Samuel Williams) * ruby -v: ruby 2.7.0dev (2019-10-19T08:19:27Z master 1d435bd51a) [x64-mingw32] * Backport: 2.5: UNKNOWN, 2.6: UNKNOWN ---------------------------------------- GH PR https://github.com/ruby/ruby/pull/2573 was a work-around that forced using some non-current MSYS2 packages libraries for building. These libraries are only available on Appveyor because their images are out-of-date, a new MSYS2 install would not have them. Using current MSYS2 libraries requires adding '-fstack-protector' to LDFLAGS. I also experimented with adding 'D_FORTIFY_SOURCE=2' to CFLAGS. Ruby build/compiles with those flags added. test-all passes. But spec tests always lock up when running: ``` C-API Thread function rb_thread_call_without_gvl - runs a C function with the global lock unlocked - runs a C function with the global lock unlocked amd unlocks IO with the generic RUBY_UBF_IO ``` The tests are in spec/ruby/optional/capi/thread_spec.rb, as shown: https://github.com/ruby/ruby/blob/1d435bd51aeca75f6e437bc92de2271a716a1ced/… and the freeze occurs on line 137. I've run this once in ruby-loco, twice in my ruby fork, and several times locally. In all runs, test-all passed, and spec stopped at the test above. This uses gcc 9.2.0, and the problem would occur with anyone doing a 'fresh' install of MSYS2. Not being a c type, this is way beyond what I can fix. Happy to try patches, etc. Thanks, Greg -- https://bugs.ruby-lang.org/

1 0

[ruby-core:113611] [Ruby master Bug#16265] Test (spec) failure using current MSYS2 tools, related to -fstack-protector and possibly D_FORTIFY_SOURCE=2
by ioquatix (Samuel Williams) 24 May '23

24 May '23

Issue #16265 has been updated by ioquatix (Samuel Williams). Okay, the root cause of the hang is because `read` in `thread_spec.c` is not `rb_w32_read`. Something is clobbering the included `#define read rb_w32_read`. ---------------------------------------- Bug #16265: Test (spec) failure using current MSYS2 tools, related to -fstack-protector and possibly D_FORTIFY_SOURCE=2 https://bugs.ruby-lang.org/issues/16265#change-103255 * Author: MSP-Greg (Greg L) * Status: Open * Priority: Normal * Assignee: ioquatix (Samuel Williams) * ruby -v: ruby 2.7.0dev (2019-10-19T08:19:27Z master 1d435bd51a) [x64-mingw32] * Backport: 2.5: UNKNOWN, 2.6: UNKNOWN ---------------------------------------- GH PR https://github.com/ruby/ruby/pull/2573 was a work-around that forced using some non-current MSYS2 packages libraries for building. These libraries are only available on Appveyor because their images are out-of-date, a new MSYS2 install would not have them. Using current MSYS2 libraries requires adding '-fstack-protector' to LDFLAGS. I also experimented with adding 'D_FORTIFY_SOURCE=2' to CFLAGS. Ruby build/compiles with those flags added. test-all passes. But spec tests always lock up when running: ``` C-API Thread function rb_thread_call_without_gvl - runs a C function with the global lock unlocked - runs a C function with the global lock unlocked amd unlocks IO with the generic RUBY_UBF_IO ``` The tests are in spec/ruby/optional/capi/thread_spec.rb, as shown: https://github.com/ruby/ruby/blob/1d435bd51aeca75f6e437bc92de2271a716a1ced/… and the freeze occurs on line 137. I've run this once in ruby-loco, twice in my ruby fork, and several times locally. In all runs, test-all passed, and spec stopped at the test above. This uses gcc 9.2.0, and the problem would occur with anyone doing a 'fresh' install of MSYS2. Not being a c type, this is way beyond what I can fix. Happy to try patches, etc. Thanks, Greg -- https://bugs.ruby-lang.org/

1 0

[ruby-core:113606] [Ruby master Bug#16265] Test (spec) failure using current MSYS2 tools, related to -fstack-protector and possibly D_FORTIFY_SOURCE=2
by ioquatix (Samuel Williams) 24 May '23

24 May '23

Issue #16265 has been updated by ioquatix (Samuel Williams). Assignee set to ioquatix (Samuel Williams) ---------------------------------------- Bug #16265: Test (spec) failure using current MSYS2 tools, related to -fstack-protector and possibly D_FORTIFY_SOURCE=2 https://bugs.ruby-lang.org/issues/16265#change-103249 * Author: MSP-Greg (Greg L) * Status: Open * Priority: Normal * Assignee: ioquatix (Samuel Williams) * ruby -v: ruby 2.7.0dev (2019-10-19T08:19:27Z master 1d435bd51a) [x64-mingw32] * Backport: 2.5: UNKNOWN, 2.6: UNKNOWN ---------------------------------------- GH PR https://github.com/ruby/ruby/pull/2573 was a work-around that forced using some non-current MSYS2 packages libraries for building. These libraries are only available on Appveyor because their images are out-of-date, a new MSYS2 install would not have them. Using current MSYS2 libraries requires adding '-fstack-protector' to LDFLAGS. I also experimented with adding 'D_FORTIFY_SOURCE=2' to CFLAGS. Ruby build/compiles with those flags added. test-all passes. But spec tests always lock up when running: ``` C-API Thread function rb_thread_call_without_gvl - runs a C function with the global lock unlocked - runs a C function with the global lock unlocked amd unlocks IO with the generic RUBY_UBF_IO ``` The tests are in spec/ruby/optional/capi/thread_spec.rb, as shown: https://github.com/ruby/ruby/blob/1d435bd51aeca75f6e437bc92de2271a716a1ced/… and the freeze occurs on line 137. I've run this once in ruby-loco, twice in my ruby fork, and several times locally. In all runs, test-all passed, and spec stopped at the test above. This uses gcc 9.2.0, and the problem would occur with anyone doing a 'fresh' install of MSYS2. Not being a c type, this is way beyond what I can fix. Happy to try patches, etc. Thanks, Greg -- https://bugs.ruby-lang.org/

1 0