Issue #19378 has been updated by aidog (Andi Idogawa). File windows-no-realpath-require.patch added Thanks to the new windows build docs by ioquatix, I made a test patch to check how much faster it would be if some of the repeated syscalls on the folders (c:/tmp/, c:/tmp/speedtest, gems and so on) are avoided: tzinfo: 0.8s to 0.3s gtk3: 2.8s to 2.5s (I see another similar issue inside the gem C code) ---------------------------------------- Bug #19378: Windows: Use less syscalls for faster require of big gems https://bugs.ruby-lang.org/issues/19378#change-101547 * Author: aidog (Andi Idogawa) * Status: Open * Priority: Normal * ruby -v: 3.2.0 * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Hello 🙂 ## Problem require is slow on windows for big gems. (example: require 'gtk3'=> 3 seconds+). This is a problem for people who want to make cross platform GUI apps with ruby. ## Possible Reason As touched on in [#15797](https://bugs.ruby-lang.org/issues/15797) it seems like require uses realpath, which is emulated on windows. It checks every parent directory. The same syscalls run many times. ## Testfile C:\tmp\speedtest\testrequire.rb: ``` ruby require __dir__ + "/helloworld1.rb" require __dir__ + "/helloworld2.rb" ``` ``` shell ruby --disable-gems C:\tmp\speedtest\testrequire.rb ``` ### Syscalls per File/Directory: 1. CreateFile 2. QueryInformationVolume 3. QueryIdInformation 4. QueryAllInformationFile 5. QueryNameInformationFile 6. QueryNameInformationFile 7. QueryNormalizedNameInformationFile 8. CloseFile ### Files/Directories checked 1. C:\tmp 2. C:\tmp\speedtest 3. C:\tmp\speedtest\helloworld1.rb 4. C:\tmp 5. C:\tmp\speedtest 6. C:\tmp\speedtest\helloworld2.rb For two required files Ruby had to do 8*6 = **48** syscalls. The syscalls orginate from rb_w32_reparse_symlink_p / lstat Rubygems live in subfolders with 9+ parts: "C:\Ruby32-x64\lib\ruby\gems\3.2.0\gems\glib2-4.0.8\lib\glib2\variant.rb" Each file takes 8 * 9 = **72**+ calls. For variant.rb it is **80** calls. The result for the syscalls don't change in such a short time, so it should be possible to cache it. With require_relative it's twice as many calls. ## Other testcases Same result: ``` ruby File.realpath __dir__ + "/helloworld1.rb" File.realpath __dir__ + "/helloworld2.rb" ``` ``` ruby File.stat __dir__ + "/helloworld1.rb" File.stat __dir__ + "/helloworld2.rb" ``` It does not happen in $LOAD_PATH.resolve_feature_path(__dir__ + "/helloworld1.rb") ## Request Would it be possible to cache the stat calls when using require? I tried to implement a cache inside the ruby source code, but failed. If not, is there now a way to combine ruby files into one? I previously talked about require here: [YJIT: Windows support lacking.](https://bugs.ruby-lang.org/issues/19325#note-11) ## How to reproduce Ruby versions: At least 3.0+, most likely older ones too. Tested using Ruby Installer 3.1 and 3.2. [Procmon Software by Sysinternals](https://learn.microsoft.com/en-us/sysinternals/downloads/procmon) ---Files-------------------------------- windows-no-realpath-require.patch (992 Bytes) -- https://bugs.ruby-lang.org/

Issue #19378 has been updated by joshc (Josh C). File windows-revert-79a4484a.patch added I've also noticed a significant increase in file IO events (as reported by procmon) due to https://github.com/ruby/ruby/commit/79a4484a072e9769b603e7b4fbdb15b1d7eccb15 introduced in Ruby 3.1.0. The code tries to prevent the same file from being loaded twice by calling `rb_realpath_internal` to see if the realpath has already been loaded. This is a problem on systems like Windows that use Ruby's emulated realpath, especially when there are deeply nested directories. I've attached a revert patch. It'd be great to use GetFinalPathNameByHandleW and avoid the emulate code. ---------------------------------------- Bug #19378: Windows: Use less syscalls for faster require of big gems https://bugs.ruby-lang.org/issues/19378#change-102015 * Author: aidog (Andi Idogawa) * Status: Assigned * Priority: Normal * Assignee: windows * ruby -v: 3.2.0 * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Hello 🙂 ## Problem require is slow on windows for big gems. (example: require 'gtk3'=> 3 seconds+). This is a problem for people who want to make cross platform GUI apps with ruby. ## Possible Reason As touched on in [#15797](https://bugs.ruby-lang.org/issues/15797) it seems like require uses realpath, which is emulated on windows. It checks every parent directory. The same syscalls run many times. ## Testfile C:\tmp\speedtest\testrequire.rb: ``` ruby require __dir__ + "/helloworld1.rb" require __dir__ + "/helloworld2.rb" ``` ``` shell ruby --disable-gems C:\tmp\speedtest\testrequire.rb ``` ### Syscalls per File/Directory: 1. CreateFile 2. QueryInformationVolume 3. QueryIdInformation 4. QueryAllInformationFile 5. QueryNameInformationFile 6. QueryNameInformationFile 7. QueryNormalizedNameInformationFile 8. CloseFile ### Files/Directories checked 1. C:\tmp 2. C:\tmp\speedtest 3. C:\tmp\speedtest\helloworld1.rb 4. C:\tmp 5. C:\tmp\speedtest 6. C:\tmp\speedtest\helloworld2.rb For two required files Ruby had to do 8*6 = **48** syscalls. The syscalls orginate from rb_w32_reparse_symlink_p / lstat Rubygems live in subfolders with 9+ parts: "C:\Ruby32-x64\lib\ruby\gems\3.2.0\gems\glib2-4.0.8\lib\glib2\variant.rb" Each file takes 8 * 9 = **72**+ calls. For variant.rb it is **80** calls. The result for the syscalls don't change in such a short time, so it should be possible to cache it. With require_relative it's twice as many calls. ## Other testcases Same result: ``` ruby File.realpath __dir__ + "/helloworld1.rb" File.realpath __dir__ + "/helloworld2.rb" ``` ``` ruby File.stat __dir__ + "/helloworld1.rb" File.stat __dir__ + "/helloworld2.rb" ``` It does not happen in $LOAD_PATH.resolve_feature_path(__dir__ + "/helloworld1.rb") ## Request Would it be possible to cache the stat calls when using require? I tried to implement a cache inside the ruby source code, but failed. If not, is there now a way to combine ruby files into one? I previously talked about require here: [YJIT: Windows support lacking.](https://bugs.ruby-lang.org/issues/19325#note-11) ## How to reproduce Ruby versions: At least 3.0+, most likely older ones too. Tested using Ruby Installer 3.1 and 3.2. [Procmon Software by Sysinternals](https://learn.microsoft.com/en-us/sysinternals/downloads/procmon) ---Files-------------------------------- windows-no-realpath-require.patch (992 Bytes) windows-revert-79a4484a.patch (5.42 KB) -- https://bugs.ruby-lang.org/

Issue #19378 has been updated by MSP-Greg (Greg L). Code using `GetFinalPathNameByHandleW` already exists in win32/win32.c, see https://github.com/ruby/ruby/blob/c43fbe4ebd2b519601f0b90ca98fa096799d3846/w... For cross-reference, see also [Bug #19246 'Rebuilding the loaded feature index much slower in Ruby 3.1'](https://bugs.ruby-lang.org/issues/19246) ---------------------------------------- Bug #19378: Windows: Use less syscalls for faster require of big gems https://bugs.ruby-lang.org/issues/19378#change-102083 * Author: aidog (Andi Idogawa) * Status: Assigned * Priority: Normal * Assignee: windows * ruby -v: 3.2.0 * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Hello 🙂 ## Problem require is slow on windows for big gems. (example: require 'gtk3'=> 3 seconds+). This is a problem for people who want to make cross platform GUI apps with ruby. ## Possible Reason As touched on in [#15797](https://bugs.ruby-lang.org/issues/15797) it seems like require uses realpath, which is emulated on windows. It checks every parent directory. The same syscalls run many times. ## Testfile C:\tmp\speedtest\testrequire.rb: ``` ruby require __dir__ + "/helloworld1.rb" require __dir__ + "/helloworld2.rb" ``` ``` shell ruby --disable-gems C:\tmp\speedtest\testrequire.rb ``` ### Syscalls per File/Directory: 1. CreateFile 2. QueryInformationVolume 3. QueryIdInformation 4. QueryAllInformationFile 5. QueryNameInformationFile 6. QueryNameInformationFile 7. QueryNormalizedNameInformationFile 8. CloseFile ### Files/Directories checked 1. C:\tmp 2. C:\tmp\speedtest 3. C:\tmp\speedtest\helloworld1.rb 4. C:\tmp 5. C:\tmp\speedtest 6. C:\tmp\speedtest\helloworld2.rb For two required files Ruby had to do 8*6 = **48** syscalls. The syscalls orginate from rb_w32_reparse_symlink_p / lstat Rubygems live in subfolders with 9+ parts: "C:\Ruby32-x64\lib\ruby\gems\3.2.0\gems\glib2-4.0.8\lib\glib2\variant.rb" Each file takes 8 * 9 = **72**+ calls. For variant.rb it is **80** calls. The result for the syscalls don't change in such a short time, so it should be possible to cache it. With require_relative it's twice as many calls. ## Other testcases Same result: ``` ruby File.realpath __dir__ + "/helloworld1.rb" File.realpath __dir__ + "/helloworld2.rb" ``` ``` ruby File.stat __dir__ + "/helloworld1.rb" File.stat __dir__ + "/helloworld2.rb" ``` It does not happen in $LOAD_PATH.resolve_feature_path(__dir__ + "/helloworld1.rb") ## Request Would it be possible to cache the stat calls when using require? I tried to implement a cache inside the ruby source code, but failed. If not, is there now a way to combine ruby files into one? I previously talked about require here: [YJIT: Windows support lacking.](https://bugs.ruby-lang.org/issues/19325#note-11) ## How to reproduce Ruby versions: At least 3.0+, most likely older ones too. Tested using Ruby Installer 3.1 and 3.2. [Procmon Software by Sysinternals](https://learn.microsoft.com/en-us/sysinternals/downloads/procmon) ---Files-------------------------------- windows-no-realpath-require.patch (992 Bytes) windows-revert-79a4484a.patch (5.42 KB) -- https://bugs.ruby-lang.org/

Issue #19378 has been updated by rlam (Robert Lam). Hello, thank you for your attention to this issue. We have a large Ruby project using windows and after upgrading from v2.1.4 to v3.4.7 we are noticing the slowness with `require`. This is not an issue under v2.1.4. Looking at the syscalls the major difference between the two versions is instead of all 8 syscalls per file \ dir as you mentioned above it's only these 3: 1. CreateFile 2. QueryNetworkOpenInformationFile 3. CloseFile For `require bigdecimal` we experience ~12000 syscalls under Ruby 3 and only ~3000 under Ruby 2. Thanks ---------------------------------------- Bug #19378: Windows: Use less syscalls for faster require of big gems https://bugs.ruby-lang.org/issues/19378#change-116040 * Author: aidog (Andi Idogawa) * Status: Assigned * Assignee: windows * ruby -v: 3.2.0 * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Hello 🙂 ## Problem require is slow on windows for big gems. (example: require 'gtk3'=> 3 seconds+). This is a problem for people who want to make cross platform GUI apps with ruby. ## Possible Reason As touched on in [#15797](https://bugs.ruby-lang.org/issues/15797) it seems like require uses realpath, which is emulated on windows. It checks every parent directory. The same syscalls run many times. ## Testfile C:\tmp\speedtest\testrequire.rb: ``` ruby require __dir__ + "/helloworld1.rb" require __dir__ + "/helloworld2.rb" ``` ``` shell ruby --disable-gems C:\tmp\speedtest\testrequire.rb ``` ### Syscalls per File/Directory: 1. CreateFile 2. QueryInformationVolume 3. QueryIdInformation 4. QueryAllInformationFile 5. QueryNameInformationFile 6. QueryNameInformationFile 7. QueryNormalizedNameInformationFile 8. CloseFile ### Files/Directories checked 1. C:\tmp 2. C:\tmp\speedtest 3. C:\tmp\speedtest\helloworld1.rb 4. C:\tmp 5. C:\tmp\speedtest 6. C:\tmp\speedtest\helloworld2.rb For two required files Ruby had to do 8*6 = **48** syscalls. The syscalls orginate from rb_w32_reparse_symlink_p / lstat Rubygems live in subfolders with 9+ parts: "C:\Ruby32-x64\lib\ruby\gems\3.2.0\gems\glib2-4.0.8\lib\glib2\variant.rb" Each file takes 8 * 9 = **72**+ calls. For variant.rb it is **80** calls. The result for the syscalls don't change in such a short time, so it should be possible to cache it. With require_relative it's twice as many calls. ## Other testcases Same result: ``` ruby File.realpath __dir__ + "/helloworld1.rb" File.realpath __dir__ + "/helloworld2.rb" ``` ``` ruby File.stat __dir__ + "/helloworld1.rb" File.stat __dir__ + "/helloworld2.rb" ``` It does not happen in $LOAD_PATH.resolve_feature_path(__dir__ + "/helloworld1.rb") ## Request Would it be possible to cache the stat calls when using require? I tried to implement a cache inside the ruby source code, but failed. If not, is there now a way to combine ruby files into one? I previously talked about require here: [YJIT: Windows support lacking.](https://bugs.ruby-lang.org/issues/19325#note-11) ## How to reproduce Ruby versions: At least 3.0+, most likely older ones too. Tested using Ruby Installer 3.1 and 3.2. [Procmon Software by Sysinternals](https://learn.microsoft.com/en-us/sysinternals/downloads/procmon) ---Files-------------------------------- windows-no-realpath-require.patch (992 Bytes) windows-revert-79a4484a.patch (5.42 KB) -- https://bugs.ruby-lang.org/

Issue #19378 has been updated by nobu (Nobuyoshi Nakada). @aidog Thank you for the patch, could you send it as a PR? ---------------------------------------- Bug #19378: Windows: Use less syscalls for faster require of big gems https://bugs.ruby-lang.org/issues/19378#change-118252 * Author: aidog (Andi Idogawa) * Status: Assigned * Assignee: windows * ruby -v: 4.0.5 * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Hello 🙂 ## Problem require is slow on windows for big gems. (example: require 'gtk3'=> 3 seconds+). This is a problem for people who want to make cross platform GUI apps with ruby. ## Possible Reason As touched on in [#15797](https://bugs.ruby-lang.org/issues/15797) it seems like require uses realpath, which is emulated on windows. It checks every parent directory. The same syscalls run many times. ## Testfile C:\tmp\speedtest\testrequire.rb: ``` ruby require __dir__ + "/helloworld1.rb" require __dir__ + "/helloworld2.rb" ``` ``` shell ruby --disable-gems C:\tmp\speedtest\testrequire.rb ``` ### Syscalls per File/Directory: 1. CreateFile 2. QueryInformationVolume 3. QueryIdInformation 4. QueryAllInformationFile 5. QueryNameInformationFile 6. QueryNameInformationFile 7. QueryNormalizedNameInformationFile 8. CloseFile ### Files/Directories checked 1. C:\tmp 2. C:\tmp\speedtest 3. C:\tmp\speedtest\helloworld1.rb 4. C:\tmp 5. C:\tmp\speedtest 6. C:\tmp\speedtest\helloworld2.rb For two required files Ruby had to do 8*6 = **48** syscalls. The syscalls orginate from rb_w32_reparse_symlink_p / lstat Rubygems live in subfolders with 9+ parts: "C:\Ruby32-x64\lib\ruby\gems\3.2.0\gems\glib2-4.0.8\lib\glib2\variant.rb" Each file takes 8 * 9 = **72**+ calls. For variant.rb it is **80** calls. The result for the syscalls don't change in such a short time, so it should be possible to cache it. With require_relative it's twice as many calls. ## Other testcases Same result: ``` ruby File.realpath __dir__ + "/helloworld1.rb" File.realpath __dir__ + "/helloworld2.rb" ``` ``` ruby File.stat __dir__ + "/helloworld1.rb" File.stat __dir__ + "/helloworld2.rb" ``` It does not happen in $LOAD_PATH.resolve_feature_path(__dir__ + "/helloworld1.rb") ## Request Would it be possible to cache the stat calls when using require? I tried to implement a cache inside the ruby source code, but failed. If not, is there now a way to combine ruby files into one? I previously talked about require here: [YJIT: Windows support lacking.](https://bugs.ruby-lang.org/issues/19325#note-11) ## How to reproduce Ruby versions: At least 3.0+, most likely older ones too. Tested using Ruby Installer 3.1 and 3.2. [Procmon Software by Sysinternals](https://learn.microsoft.com/en-us/sysinternals/downloads/procmon) ---Files-------------------------------- windows-no-realpath-require.patch (992 Bytes) windows-revert-79a4484a.patch (5.42 KB) MINIMAL-stat-fastpath.patch (6.41 KB) recommendations.txt (4.01 KB) -- https://bugs.ruby-lang.org/