[ruby-core:114804] [Ruby master Bug#19890] File#realine(chomp: true) slower/more allocations than readline.chomp!

19 Sep 2023

      Issue #19890 has been updated by tenderlovemaking (Aaron Patterson).

This is an implementation detail, but IO#readline is [implemented as a C function](https://github.com/tenderlove/ruby/blob/2334570c8f18d8f2fca3d1d947853e30f7e1...), and currently there is no way to pass keyword args to a C function without allocating a hash.  I re-implemeted IO#readline in Ruby and it does eliminate the allocation overhead (I've not tested performance).

I sent the patch [here](https://github.com/ruby/ruby/pull/8473).

More implementation details, but the challenge with this patch is that `IO#readline` sets `$_` to the last read line, but it does that only in the caller's frame.  Take the following program as an example:

```ruby
class Foo
  def call
    File.open(__FILE__) do |f|
      read f
      p __method__ => $_
    end
  end

  def read f
    f.readline
    p __method__ => $_
  end
end

Foo.new.call
```

If you run this, the output is:

```
$ ruby -v test.rb
ruby 3.2.2 (2023-03-30 revision e51014f9c0) [arm64-darwin22]
{:read=>"class Foo\n"}
{:call=>nil}
```

So `$_` looks like a global, but it's not really.  `$_` is set in the `read` method, but not set in the `call` method.

The callee (`IO#readline`) is manipulating values _in the caller's_ environment!  The last line value is written via [`rb_lastline_set`](https://github.com/tenderlove/ruby/blob/501aeb3432fd1ed4b4827841b287bf7a44a3...).  This function eventually [walks up the call stack, looking for the most recent Ruby call frame](https://github.com/tenderlove/ruby/blob/501aeb3432fd1ed4b4827841b287bf7a44a3...) and sets `$_` in that frame.  Since `IO#readline` was implemented in C, "the most recent Ruby call frame" is the user code (in my example `Foo#read`).

However, moving `IO#readline` to Ruby means that "the most recent Ruby call frame" is `IO#readline` itself.  Of course, setting `$_` in `IO#readline` is of no use to anyone, so in my PR I [added a function that that lets us set special variables in other frames](https://github.com/ruby/ruby/pull/8473/files#diff-2af2e7f2e1c28da5e9d99ad117...).  I'm not particularly thrilled with this because it's coupling the implementation of `IO#readline` with its stack depth.  That said, it supports `$_` and fixes this issue.

I think it would be cool if we could push a special frame for methods like `IO#readline` so that we know where user code is, but I feel a solution like that is beyond the scope of this ticket.

----------------------------------------
Bug #19890: File#realine(chomp: true) slower/more allocations than readline.chomp!
https://bugs.ruby-lang.org/issues/19890#change-104648

* Author: segiddins (Samuel Giddins)
* Status: Open
* Priority: Normal
* ruby -v: 3.2.2
* Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN
----------------------------------------
On ruby 3.2.2 running the following script:

``` ruby
#!/usr/bin/env ruby

require 'rubygems'
require 'bundler/inline'

puts RUBY_VERSION

gemfile do
  source "https://rubygems.org"
  gem "benchmark-ipsa"
end

Benchmark.ipsa do |x|
  x.report("f.readline(chomp: true)") do
    File.open("/usr/share/dict/words") do |f|
      f.readline(chomp: true) until f.eof?
    end
  end

  x.report("f.readline.chomp!") do
    File.open("/usr/share/dict/words") do |f|
      until f.eof?
        s = f.readline
        s.chomp!
        s
      end
    end
  end

  x.report("f.readline.chomp") do
    File.open("/usr/share/dict/words") do |f|
      until f.eof?
        f.readline.chomp
      end
    end
  end

  x.compare!
end
```

I get the following (surprising) result:

```
3.2.2
Allocations -------------------------------------
	f.readline(chomp: true)
                      707931/1  alloc/ret       50/1  strings/ret
   f.readline.chomp!  235979/1  alloc/ret       50/1  strings/ret
    f.readline.chomp  471955/1  alloc/ret       50/1  strings/ret
Warming up --------------------------------------
f.readline(chomp: true)
                         1.000  i/100ms
   f.readline.chomp!     2.000  i/100ms
    f.readline.chomp     2.000  i/100ms
Calculating -------------------------------------
f.readline(chomp: true)
                         16.165  (± 6.2%) i/s -     81.000 
   f.readline.chomp!     25.246  (± 7.9%) i/s -    126.000 
    f.readline.chomp     20.997  (± 9.5%) i/s -    106.000 

Comparison:
   f.readline.chomp!:       25.2 i/s
    f.readline.chomp:       21.0 i/s - 1.20x slower
f.readline(chomp: true):       16.2 i/s - 1.56x slower
```

I would expect `File#readline(chomp: true)` to be comparable to `s = f.readline; s.chomp!; s` at a bare minimum, but it is slower and has more allocations even than `readline.chomp`

-- 
https://bugs.ruby-lang.org/