[ruby-core:120336] [Ruby master Bug#20972] OpenSSL Memory Usage
Issue #20972 has been reported by mrkgrandjean (Mark Grandjean). ---------------------------------------- Bug #20972: OpenSSL Memory Usage https://bugs.ruby-lang.org/issues/20972 * Author: mrkgrandjean (Mark Grandjean) * Status: Open * ruby -v: 3.3.6 * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- While testing large file uploads I noticed the OpenSSL Buffering is allocating a lot of memory for file uploads. Similar to the issue in this ticket from a few years ago https://bugs.ruby-lang.org/issues/14426 Using the same test code from that ticket ``` ruby require "http" require "memory_profiler" require "stringio" body = StringIO.new("a" * 5*1024*1024) MemoryProfiler.report do HTTP.post("https://example.com", body: body) end.pretty_print ``` in ruby 2.6.10 ``` allocated memory by gem ----------------------------------- 1913287 unicode_normalize 1053463 llhttp-ffi-0.5.0 187849 http-5.2.0 50168 openssl 36180 addressable-2.8.7 8904 ipaddr 272 other ``` While testing in Ruby 3.3.6 ``` allocated memory by gem ----------------------------------- 16104976 openssl 2393456 unicode_normalize 1053880 llhttp-ffi-0.5.0 151633 http-5.2.0 9813 addressable-2.8.7 7872 ipaddr 240 other 40 forwardable ``` It looks like the memory allocation issue has returned. -- https://bugs.ruby-lang.org/
Issue #20972 has been updated by luke-gru (Luke Gruber). I took a look at this and this is due to 2 things that I could notice: The call to `string.b` if it's not encoded as binary already, in the `Buffer` class in `openssl/buffering.rb`. The other one is a call to `rb_str_new_frozen` in `ossl_ssl_write_internal`. Getting rid of the `rb_str_new_frozen` call brings the allocated memory down to ~ `5639000` and removing the other one gets it down to `42840`. I'll leave it to someone with more familiarity with the openssl codebase to provide a good fix, if it's possible, possibly using `locktmp` or something for the frozen case. Maybe it's also possible to just force the encoding as binary for the provided string instead of duplicating it. ---------------------------------------- Bug #20972: OpenSSL Memory Usage https://bugs.ruby-lang.org/issues/20972#change-111132 * Author: mrkgrandjean (Mark Grandjean) * Status: Open * ruby -v: 3.3.6 * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- While testing large file uploads I noticed the OpenSSL Buffering is allocating a lot of memory for file uploads. Similar to the issue in this ticket from a few years ago https://bugs.ruby-lang.org/issues/14426 Using the same test code from that ticket ``` ruby require "http" require "memory_profiler" require "stringio" body = StringIO.new("a" * 5*1024*1024) MemoryProfiler.report do HTTP.post("https://example.com", body: body) end.pretty_print ``` in ruby 2.6.10 ``` allocated memory by gem ----------------------------------- 1913287 unicode_normalize 1053463 llhttp-ffi-0.5.0 187849 http-5.2.0 50168 openssl 36180 addressable-2.8.7 8904 ipaddr 272 other ``` While testing in Ruby 3.3.6 ``` allocated memory by gem ----------------------------------- 16104976 openssl 2393456 unicode_normalize 1053880 llhttp-ffi-0.5.0 151633 http-5.2.0 9813 addressable-2.8.7 7872 ipaddr 240 other 40 forwardable ``` It looks like the memory allocation issue has returned. -- https://bugs.ruby-lang.org/
Issue #20972 has been updated by byroot (Jean Boussier). I think you are mostly correct, what exactly cause the memory growth is hard to point precisely (especially with `memory_profiler` because this codepath depends a lot on shared strings, so it's not fully clear which call exactly cause the unsharing. But you are right that using `rb_str_locktmp` would avoid some allocations and probably some copying, and there are a few other small changes that can be made to rely less on string sharing, and reduce allocations a bit: https://github.com/ruby/openssl/pull/831 ---------------------------------------- Bug #20972: OpenSSL Memory Usage https://bugs.ruby-lang.org/issues/20972#change-111134 * Author: mrkgrandjean (Mark Grandjean) * Status: Open * ruby -v: 3.3.6 * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- While testing large file uploads I noticed the OpenSSL Buffering is allocating a lot of memory for file uploads. Similar to the issue in this ticket from a few years ago https://bugs.ruby-lang.org/issues/14426 Using the same test code from that ticket ``` ruby require "http" require "memory_profiler" require "stringio" body = StringIO.new("a" * 5*1024*1024) MemoryProfiler.report do HTTP.post("https://example.com", body: body) end.pretty_print ``` in ruby 2.6.10 ``` allocated memory by gem ----------------------------------- 1913287 unicode_normalize 1053463 llhttp-ffi-0.5.0 187849 http-5.2.0 50168 openssl 36180 addressable-2.8.7 8904 ipaddr 272 other ``` While testing in Ruby 3.3.6 ``` allocated memory by gem ----------------------------------- 16104976 openssl 2393456 unicode_normalize 1053880 llhttp-ffi-0.5.0 151633 http-5.2.0 9813 addressable-2.8.7 7872 ipaddr 240 other 40 forwardable ``` It looks like the memory allocation issue has returned. -- https://bugs.ruby-lang.org/
participants (3)
-
byroot (Jean Boussier) -
luke-gru (Luke Gruber)
-
mrkgrandjean (Mark Grandjean)