[ruby-core:115304] [Ruby master Feature#19993] Optionally Free all memory at exit
Issue #19993 has been reported by HParker (Adam Hess). ---------------------------------------- Feature #19993: Optionally Free all memory at exit https://bugs.ruby-lang.org/issues/19993 * Author: HParker (Adam Hess) * Status: Open * Priority: Normal ---------------------------------------- Add a runtime option allowing Ruby to optionally free all memory at shutdown. # why Today, memory sanitizers are difficult to use with Ruby, since not all memory is freed at shutdown. it is difficult to detect memory leaks or errors in Ruby or in Ruby C extensions when these tools are not available. While implementing this feature, we were able to identify and fix a number of memory leaks and errors. https://github.com/ruby/ruby/pull/8556 https://github.com/ruby/ruby/pull/8512 https://github.com/ruby/ruby/pull/8503 https://github.com/ruby/ruby/pull/8487 https://github.com/ruby/ruby/pull/8452 https://github.com/ruby/ruby/pull/8501 https://github.com/ruby/ruby/pull/8481 https://github.com/ruby/ruby/pull/8555 This shows that having access to tools like this can make finding and fixing memory bugs easier. # current progress Today we can allow ruby developers to enable freeing memory at shutdown via the `free-at-shutdown` runtime option. PR: https://github.com/ruby/ruby/pull/8868 running, without `--free-on-shutdown`: ``` valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes -- ./miniruby basictest/test.rb ``` ``` ==573270== LEAK SUMMARY: ==573270== definitely lost: 658,324 bytes in 5,387 blocks ==573270== indirectly lost: 955,708 bytes in 11,957 blocks ==573270== possibly lost: 2,071,096 bytes in 12 blocks ==573270== still reachable: 161,881 bytes in 275 blocks ==573270== suppressed: 0 bytes in 0 blocks ==573270== Reachable blocks (those to which a pointer was found) are not shown. ==573270== To see them, rerun with: --leak-check=full --show-leak-kinds=all ``` with `--free-on-shutdown` ``` valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes -- ./miniruby --free-on-shutdown basictest/test.rb ``` ``` ==573306== HEAP SUMMARY: ==573306== in use at exit: 0 bytes in 0 blocks ==573306== total heap usage: 43,643 allocs, 43,643 frees, 29,222,534 bytes allocated ==573306== ==573306== All heap blocks were freed -- no leaks are possible ``` # future plans * Continue improving memory safety * add `valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes -- ./miniruby --free-on-shutdown basictest/test.rb` to CI * Allow C extensions to do a "optional destruct for memory safety" so they can leverage the same memory sanitizer tools. -- https://bugs.ruby-lang.org/
Issue #19993 has been updated by mame (Yusuke Endoh). Nice job! It's also very good that they actually fix a lot of memory leak bugs. Thanks. The reason for making it optional is to prevent an increase in the exit time? I wonder if it is worth the complexity of adding the command-line option. I concern the increase in footprint a little, however, the size of ruby executable become 45,949,416 bytes to 45,963,736 bytes in my environment (0.03% increase). I don't think this is a problem. ---------------------------------------- Feature #19993: Optionally Free all memory at exit https://bugs.ruby-lang.org/issues/19993#change-105223 * Author: HParker (Adam Hess) * Status: Open * Priority: Normal ---------------------------------------- Add a runtime option allowing Ruby to optionally free all memory at shutdown. # why Today, memory sanitizers are difficult to use with Ruby, since not all memory is freed at shutdown. it is difficult to detect memory leaks or errors in Ruby or in Ruby C extensions when these tools are not available. While implementing this feature, we were able to identify and fix a number of memory leaks and errors. https://github.com/ruby/ruby/pull/8556 https://github.com/ruby/ruby/pull/8512 https://github.com/ruby/ruby/pull/8503 https://github.com/ruby/ruby/pull/8487 https://github.com/ruby/ruby/pull/8452 https://github.com/ruby/ruby/pull/8501 https://github.com/ruby/ruby/pull/8481 https://github.com/ruby/ruby/pull/8555 This shows that having access to tools like this can make finding and fixing memory bugs easier. # current progress Today we can allow ruby developers to enable freeing memory at shutdown via the `free-at-shutdown` runtime option. PR: https://github.com/ruby/ruby/pull/8868 running, without `--free-on-shutdown`: ``` valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes -- ./miniruby basictest/test.rb ``` ``` ==573270== LEAK SUMMARY: ==573270== definitely lost: 658,324 bytes in 5,387 blocks ==573270== indirectly lost: 955,708 bytes in 11,957 blocks ==573270== possibly lost: 2,071,096 bytes in 12 blocks ==573270== still reachable: 161,881 bytes in 275 blocks ==573270== suppressed: 0 bytes in 0 blocks ==573270== Reachable blocks (those to which a pointer was found) are not shown. ==573270== To see them, rerun with: --leak-check=full --show-leak-kinds=all ``` with `--free-on-shutdown` ``` valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes -- ./miniruby --free-on-shutdown basictest/test.rb ``` ``` ==573306== HEAP SUMMARY: ==573306== in use at exit: 0 bytes in 0 blocks ==573306== total heap usage: 43,643 allocs, 43,643 frees, 29,222,534 bytes allocated ==573306== ==573306== All heap blocks were freed -- no leaks are possible ``` # future plans * Continue improving memory safety * add `valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes -- ./miniruby --free-on-shutdown basictest/test.rb` to CI * Allow C extensions to do a "optional destruct for memory safety" so they can leverage the same memory sanitizer tools. -- https://bugs.ruby-lang.org/
Issue #19993 has been updated by HParker (Adam Hess).
The reason for making it optional is to prevent an increase in the exit time? I wonder if it is worth the complexity of adding the command-line option.
Yeah, I don't know how concerned about exit time I should be, but if you aren't testing memory behavior, I am not sure if there is a compelling reason to free memory that that is sure to not grow. I do agree it would be simpler to skip the runtime flag though. ---------------------------------------- Feature #19993: Optionally Free all memory at exit https://bugs.ruby-lang.org/issues/19993#change-105231 * Author: HParker (Adam Hess) * Status: Open * Priority: Normal ---------------------------------------- Add a runtime option allowing Ruby to optionally free all memory at shutdown. # why Today, memory sanitizers are difficult to use with Ruby, since not all memory is freed at shutdown. it is difficult to detect memory leaks or errors in Ruby or in Ruby C extensions when these tools are not available. While implementing this feature, we were able to identify and fix a number of memory leaks and errors. https://github.com/ruby/ruby/pull/8556 https://github.com/ruby/ruby/pull/8512 https://github.com/ruby/ruby/pull/8503 https://github.com/ruby/ruby/pull/8487 https://github.com/ruby/ruby/pull/8452 https://github.com/ruby/ruby/pull/8501 https://github.com/ruby/ruby/pull/8481 https://github.com/ruby/ruby/pull/8555 This shows that having access to tools like this can make finding and fixing memory bugs easier. # current progress Today we can allow ruby developers to enable freeing memory at shutdown via the `free-at-shutdown` runtime option. PR: https://github.com/ruby/ruby/pull/8868 running, without `--free-on-shutdown`: ``` valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes -- ./miniruby basictest/test.rb ``` ``` ==573270== LEAK SUMMARY: ==573270== definitely lost: 658,324 bytes in 5,387 blocks ==573270== indirectly lost: 955,708 bytes in 11,957 blocks ==573270== possibly lost: 2,071,096 bytes in 12 blocks ==573270== still reachable: 161,881 bytes in 275 blocks ==573270== suppressed: 0 bytes in 0 blocks ==573270== Reachable blocks (those to which a pointer was found) are not shown. ==573270== To see them, rerun with: --leak-check=full --show-leak-kinds=all ``` with `--free-on-shutdown` ``` valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes -- ./miniruby --free-on-shutdown basictest/test.rb ``` ``` ==573306== HEAP SUMMARY: ==573306== in use at exit: 0 bytes in 0 blocks ==573306== total heap usage: 43,643 allocs, 43,643 frees, 29,222,534 bytes allocated ==573306== ==573306== All heap blocks were freed -- no leaks are possible ``` # future plans * Continue improving memory safety * add `valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes -- ./miniruby --free-on-shutdown basictest/test.rb` to CI * Allow C extensions to do a "optional destruct for memory safety" so they can leverage the same memory sanitizer tools. -- https://bugs.ruby-lang.org/
Issue #19993 has been updated by byroot (Jean Boussier).
I don't know how concerned about exit time I should be
Minor increase is acceptable, but it's quite important in several cases, e.g. command line tools, etc. Perhaps it can be an environment variable? ---------------------------------------- Feature #19993: Optionally Free all memory at exit https://bugs.ruby-lang.org/issues/19993#change-105232 * Author: HParker (Adam Hess) * Status: Open * Priority: Normal ---------------------------------------- Add a runtime option allowing Ruby to optionally free all memory at shutdown. # why Today, memory sanitizers are difficult to use with Ruby, since not all memory is freed at shutdown. it is difficult to detect memory leaks or errors in Ruby or in Ruby C extensions when these tools are not available. While implementing this feature, we were able to identify and fix a number of memory leaks and errors. https://github.com/ruby/ruby/pull/8556 https://github.com/ruby/ruby/pull/8512 https://github.com/ruby/ruby/pull/8503 https://github.com/ruby/ruby/pull/8487 https://github.com/ruby/ruby/pull/8452 https://github.com/ruby/ruby/pull/8501 https://github.com/ruby/ruby/pull/8481 https://github.com/ruby/ruby/pull/8555 This shows that having access to tools like this can make finding and fixing memory bugs easier. # current progress Today we can allow ruby developers to enable freeing memory at shutdown via the `free-at-shutdown` runtime option. PR: https://github.com/ruby/ruby/pull/8868 running, without `--free-on-shutdown`: ``` valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes -- ./miniruby basictest/test.rb ``` ``` ==573270== LEAK SUMMARY: ==573270== definitely lost: 658,324 bytes in 5,387 blocks ==573270== indirectly lost: 955,708 bytes in 11,957 blocks ==573270== possibly lost: 2,071,096 bytes in 12 blocks ==573270== still reachable: 161,881 bytes in 275 blocks ==573270== suppressed: 0 bytes in 0 blocks ==573270== Reachable blocks (those to which a pointer was found) are not shown. ==573270== To see them, rerun with: --leak-check=full --show-leak-kinds=all ``` with `--free-on-shutdown` ``` valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes -- ./miniruby --free-on-shutdown basictest/test.rb ``` ``` ==573306== HEAP SUMMARY: ==573306== in use at exit: 0 bytes in 0 blocks ==573306== total heap usage: 43,643 allocs, 43,643 frees, 29,222,534 bytes allocated ==573306== ==573306== All heap blocks were freed -- no leaks are possible ``` # future plans * Continue improving memory safety * add `valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes -- ./miniruby --free-on-shutdown basictest/test.rb` to CI * Allow C extensions to do a "optional destruct for memory safety" so they can leverage the same memory sanitizer tools. -- https://bugs.ruby-lang.org/
Issue #19993 has been updated by mame (Yusuke Endoh). I wonder what kind of micro-benchmark would have the largest overhead. Loading 100,000 empty files ``` $ time ./miniruby -e 'n = "a"; 100000.times { require "./empty-files/#{ n }"; n.succ! }' real 0m2.706s user 0m1.112s sys 0m1.567s $ time ./miniruby --free-on-shutdown -e 'n = "a"; 100000.times { require "./empty-files/#{ n }"; n.succ! }' ./miniruby: warning: Free on shutdown is experimental and may be unstable real 0m2.747s user 0m1.318s sys 0m1.402s ``` Defining 1,000,000 global variables ``` $ time ./miniruby -e 's = []; n = "a"; 1000000.times { s << "$#{ n }=1"; n.succ! }; eval(s.join(";"))' real 0m1.512s user 0m1.251s sys 0m0.260s $ time ./miniruby --free-on-shutdown -e 's = []; n = "a"; 1000000.times { s << "$#{ n }=1"; n.succ! }; eval(s.join(";"))' ./miniruby: warning: Free on shutdown is experimental and may be unstable real 0m1.628s user 0m1.352s sys 0m0.276s ``` I don't even know if the real-world cases could be ever a problem, but I agree it's safe to default off. ---------------------------------------- Feature #19993: Optionally Free all memory at exit https://bugs.ruby-lang.org/issues/19993#change-105236 * Author: HParker (Adam Hess) * Status: Open * Priority: Normal ---------------------------------------- Add a runtime option allowing Ruby to optionally free all memory at shutdown. # why Today, memory sanitizers are difficult to use with Ruby, since not all memory is freed at shutdown. it is difficult to detect memory leaks or errors in Ruby or in Ruby C extensions when these tools are not available. While implementing this feature, we were able to identify and fix a number of memory leaks and errors. https://github.com/ruby/ruby/pull/8556 https://github.com/ruby/ruby/pull/8512 https://github.com/ruby/ruby/pull/8503 https://github.com/ruby/ruby/pull/8487 https://github.com/ruby/ruby/pull/8452 https://github.com/ruby/ruby/pull/8501 https://github.com/ruby/ruby/pull/8481 https://github.com/ruby/ruby/pull/8555 This shows that having access to tools like this can make finding and fixing memory bugs easier. # current progress Today we can allow ruby developers to enable freeing memory at shutdown via the `free-at-shutdown` runtime option. PR: https://github.com/ruby/ruby/pull/8868 running, without `--free-on-shutdown`: ``` valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes -- ./miniruby basictest/test.rb ``` ``` ==573270== LEAK SUMMARY: ==573270== definitely lost: 658,324 bytes in 5,387 blocks ==573270== indirectly lost: 955,708 bytes in 11,957 blocks ==573270== possibly lost: 2,071,096 bytes in 12 blocks ==573270== still reachable: 161,881 bytes in 275 blocks ==573270== suppressed: 0 bytes in 0 blocks ==573270== Reachable blocks (those to which a pointer was found) are not shown. ==573270== To see them, rerun with: --leak-check=full --show-leak-kinds=all ``` with `--free-on-shutdown` ``` valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes -- ./miniruby --free-on-shutdown basictest/test.rb ``` ``` ==573306== HEAP SUMMARY: ==573306== in use at exit: 0 bytes in 0 blocks ==573306== total heap usage: 43,643 allocs, 43,643 frees, 29,222,534 bytes allocated ==573306== ==573306== All heap blocks were freed -- no leaks are possible ``` # future plans * Continue improving memory safety * add `valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes -- ./miniruby --free-on-shutdown basictest/test.rb` to CI * Allow C extensions to do a "optional destruct for memory safety" so they can leverage the same memory sanitizer tools. -- https://bugs.ruby-lang.org/
Issue #19993 has been updated by byroot (Jean Boussier).
I don't even know if the real-world cases could be ever a problem
Well, for one thing we are currently experiencing a (moderate) issue caused by slow exit in production. We have some code in Pitchfork that basically daemonize and is expected to be "fast" (`Process.wait(fork { fork })`), but sometimes take a handful of seconds because if incremental GC was going on, the `rb_ec_cleanup` will finish GC before exiting. In this case the numbers you show are no concern to me, but just giving an example of why quick exit is desirable in real world use cases. So I agree to default to off. The reason I suggest an environment variables (probably under `RUBY_GC_`), is that if I'm not mistaken this would be used by tools such as `ruby_memcheck`, so it's easy for these tools to inject an environment variable. Much easier than a command like flag. ---------------------------------------- Feature #19993: Optionally Free all memory at exit https://bugs.ruby-lang.org/issues/19993#change-105238 * Author: HParker (Adam Hess) * Status: Open * Priority: Normal ---------------------------------------- Add a runtime option allowing Ruby to optionally free all memory at shutdown. # why Today, memory sanitizers are difficult to use with Ruby, since not all memory is freed at shutdown. it is difficult to detect memory leaks or errors in Ruby or in Ruby C extensions when these tools are not available. While implementing this feature, we were able to identify and fix a number of memory leaks and errors. https://github.com/ruby/ruby/pull/8556 https://github.com/ruby/ruby/pull/8512 https://github.com/ruby/ruby/pull/8503 https://github.com/ruby/ruby/pull/8487 https://github.com/ruby/ruby/pull/8452 https://github.com/ruby/ruby/pull/8501 https://github.com/ruby/ruby/pull/8481 https://github.com/ruby/ruby/pull/8555 This shows that having access to tools like this can make finding and fixing memory bugs easier. # current progress Today we can allow ruby developers to enable freeing memory at shutdown via the `free-at-shutdown` runtime option. PR: https://github.com/ruby/ruby/pull/8868 running, without `--free-on-shutdown`: ``` valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes -- ./miniruby basictest/test.rb ``` ``` ==573270== LEAK SUMMARY: ==573270== definitely lost: 658,324 bytes in 5,387 blocks ==573270== indirectly lost: 955,708 bytes in 11,957 blocks ==573270== possibly lost: 2,071,096 bytes in 12 blocks ==573270== still reachable: 161,881 bytes in 275 blocks ==573270== suppressed: 0 bytes in 0 blocks ==573270== Reachable blocks (those to which a pointer was found) are not shown. ==573270== To see them, rerun with: --leak-check=full --show-leak-kinds=all ``` with `--free-on-shutdown` ``` valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes -- ./miniruby --free-on-shutdown basictest/test.rb ``` ``` ==573306== HEAP SUMMARY: ==573306== in use at exit: 0 bytes in 0 blocks ==573306== total heap usage: 43,643 allocs, 43,643 frees, 29,222,534 bytes allocated ==573306== ==573306== All heap blocks were freed -- no leaks are possible ``` # future plans * Continue improving memory safety * add `valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes -- ./miniruby --free-on-shutdown basictest/test.rb` to CI * Allow C extensions to do a "optional destruct for memory safety" so they can leverage the same memory sanitizer tools. -- https://bugs.ruby-lang.org/
Issue #19993 has been updated by HParker (Adam Hess). I can't think of a situation where an environment variable is worse then a command line option, so I am going to update the PR to just use a environment variable. It should make testing in environments like Rails apps easier. ---------------------------------------- Feature #19993: Optionally Free all memory at exit https://bugs.ruby-lang.org/issues/19993#change-105245 * Author: HParker (Adam Hess) * Status: Open * Priority: Normal ---------------------------------------- Add a runtime option allowing Ruby to optionally free all memory at shutdown. # why Today, memory sanitizers are difficult to use with Ruby, since not all memory is freed at shutdown. it is difficult to detect memory leaks or errors in Ruby or in Ruby C extensions when these tools are not available. While implementing this feature, we were able to identify and fix a number of memory leaks and errors. https://github.com/ruby/ruby/pull/8556 https://github.com/ruby/ruby/pull/8512 https://github.com/ruby/ruby/pull/8503 https://github.com/ruby/ruby/pull/8487 https://github.com/ruby/ruby/pull/8452 https://github.com/ruby/ruby/pull/8501 https://github.com/ruby/ruby/pull/8481 https://github.com/ruby/ruby/pull/8555 This shows that having access to tools like this can make finding and fixing memory bugs easier. # current progress Today we can allow ruby developers to enable freeing memory at shutdown via the `free-at-shutdown` runtime option. PR: https://github.com/ruby/ruby/pull/8868 running, without `--free-on-shutdown`: ``` valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes -- ./miniruby basictest/test.rb ``` ``` ==573270== LEAK SUMMARY: ==573270== definitely lost: 658,324 bytes in 5,387 blocks ==573270== indirectly lost: 955,708 bytes in 11,957 blocks ==573270== possibly lost: 2,071,096 bytes in 12 blocks ==573270== still reachable: 161,881 bytes in 275 blocks ==573270== suppressed: 0 bytes in 0 blocks ==573270== Reachable blocks (those to which a pointer was found) are not shown. ==573270== To see them, rerun with: --leak-check=full --show-leak-kinds=all ``` with `--free-on-shutdown` ``` valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes -- ./miniruby --free-on-shutdown basictest/test.rb ``` ``` ==573306== HEAP SUMMARY: ==573306== in use at exit: 0 bytes in 0 blocks ==573306== total heap usage: 43,643 allocs, 43,643 frees, 29,222,534 bytes allocated ==573306== ==573306== All heap blocks were freed -- no leaks are possible ``` # future plans * Continue improving memory safety * add `valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes -- ./miniruby --free-on-shutdown basictest/test.rb` to CI * Allow C extensions to do a "optional destruct for memory safety" so they can leverage the same memory sanitizer tools. -- https://bugs.ruby-lang.org/
Issue #19993 has been updated by matz (Yukihiro Matsumoto). Agreed to optionally free all memory at exit. But we haven't used the term “shutdown” for process termination. How about using `RUBY_FREE_AT_EXIT` environment variable? Matz. ---------------------------------------- Feature #19993: Optionally Free all memory at exit https://bugs.ruby-lang.org/issues/19993#change-105550 * Author: HParker (Adam Hess) * Status: Open * Priority: Normal ---------------------------------------- Add a runtime option allowing Ruby to optionally free all memory at shutdown. # why Today, memory sanitizers are difficult to use with Ruby, since not all memory is freed at shutdown. it is difficult to detect memory leaks or errors in Ruby or in Ruby C extensions when these tools are not available. While implementing this feature, we were able to identify and fix a number of memory leaks and errors. https://github.com/ruby/ruby/pull/8556 https://github.com/ruby/ruby/pull/8512 https://github.com/ruby/ruby/pull/8503 https://github.com/ruby/ruby/pull/8487 https://github.com/ruby/ruby/pull/8452 https://github.com/ruby/ruby/pull/8501 https://github.com/ruby/ruby/pull/8481 https://github.com/ruby/ruby/pull/8555 This shows that having access to tools like this can make finding and fixing memory bugs easier. # current progress Today we can allow ruby developers to enable freeing memory at shutdown via the `free-at-shutdown` runtime option. PR: https://github.com/ruby/ruby/pull/8868 running, without `--free-on-shutdown`: ``` valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes -- ./miniruby basictest/test.rb ``` ``` ==573270== LEAK SUMMARY: ==573270== definitely lost: 658,324 bytes in 5,387 blocks ==573270== indirectly lost: 955,708 bytes in 11,957 blocks ==573270== possibly lost: 2,071,096 bytes in 12 blocks ==573270== still reachable: 161,881 bytes in 275 blocks ==573270== suppressed: 0 bytes in 0 blocks ==573270== Reachable blocks (those to which a pointer was found) are not shown. ==573270== To see them, rerun with: --leak-check=full --show-leak-kinds=all ``` with `--free-on-shutdown` ``` valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes -- ./miniruby --free-on-shutdown basictest/test.rb ``` ``` ==573306== HEAP SUMMARY: ==573306== in use at exit: 0 bytes in 0 blocks ==573306== total heap usage: 43,643 allocs, 43,643 frees, 29,222,534 bytes allocated ==573306== ==573306== All heap blocks were freed -- no leaks are possible ``` # future plans * Continue improving memory safety * add `valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes -- ./miniruby --free-on-shutdown basictest/test.rb` to CI * Allow C extensions to do a "optional destruct for memory safety" so they can leverage the same memory sanitizer tools. -- https://bugs.ruby-lang.org/
participants (4)
-
byroot (Jean Boussier) -
HParker (Adam Hess)
-
mame (Yusuke Endoh)
-
matz (Yukihiro Matsumoto)