Issue #22431 has been updated by matz (Yukihiro Matsumoto). I understand the need. Reading length-prefixed messages from a stream is common, and doing it efficiently in Ruby is hard today. But I don't want to bring the whole `unpack` template to IO. As @mame pointed out, `X`, `@` and `*` assume random access to a finite string, and giving them stream semantics makes the same template behave differently on String and IO. We would also have to define what happens at EOF in the middle of a value, with nonblocking IO, and in text mode. For fixed-size data, `io.read(n).unpack` already works. The real problem is variable-length data. Could you narrow the proposal to that? It could be a method that reads one variable-length value, or a lower-level primitive that lets Ruby code do it efficiently. Matz. ---------------------------------------- Feature #22431: Add `IO#unpack` and `IO#unpack1` https://bugs.ruby-lang.org/issues/22431#change-119496 * Author: ntkme (Natsuki Natsume) * Status: Open ---------------------------------------- I would like to have ruby add `IO#unpack` and `IO#unpack1` that effectively does both `IO#read(...)` and `String#unpacking(...)` in one call. This can significantly improve the performance of consuming data in a variable-length encoding format (E.g. LEB128) from a non-peekable `IO` stream (e.g. network/stdio). In a streaming `IO` with variable-length encoding, today it requires the consumer to scan the stream byte-by-byte to precisely finding the ending of a variable-length segment (E.g. LEB128). If we can have `IO#unpack`/`IO#unpack1`, it will be able to precisely consume a variable-length segment by itself, without needing a two-step solution that requires lots of intermediate ruby object allocation. --- Here is a read world example: LEB128 pack/unpack was recently added by: https://bugs.ruby-lang.org/issues/21785 A very common use case for this is LEB128 delimited payload streaming. In real world it can be a LEB128 delimited protobuf message stream. For example: ``` length, payload, length, payload, length, payload... ``` In pseudo code, reading and writing such stream would look like: ```ruby def write(payload, io) io.write([payload.length].pack('R')) io.write(payload) end def read(io) length = io.unpack1('R') # ^ This method does not exist, which makes the real world implementation slower and more complicated. # See attached benchmark.rb for working implementations. io.read(length) end ``` --- See attached `benchmark.rb`, which demonstrates the limitation of native unpacking for LEB128: ``` ruby 4.1.0dev (2026-10-12T17:01:34Z master 98fd04f76c) +YJIT +PRISM [arm64-darwin27] Warming up -------------------------------------- write varint (pure ruby) 12.167k i/100ms write leb128 (native) 15.710k i/100ms Calculating ------------------------------------- write varint (pure ruby) 124.060k (± 1.6%) i/s (8.06 μs/i) - 620.517k in 5.001746s write leb128 (native) 157.036k (± 1.1%) i/s (6.37 μs/i) - 785.500k in 5.002040s Comparison: write leb128 (native) : 157035.9 i/s write varint (pure ruby): 124060.1 i/s - 1.27x slower ruby 4.1.0dev (2026-10-12T17:01:34Z master 98fd04f76c) +YJIT +PRISM [arm64-darwin27] Warming up -------------------------------------- read varint (pure ruby) 42.025k i/100ms read leb128 (native) 20.246k i/100ms Calculating ------------------------------------- read varint (pure ruby) 423.775k (± 0.8%) i/s (2.36 μs/i) - 2.143M in 5.057579s read leb128 (native) 201.781k (± 1.4%) i/s (4.96 μs/i) - 1.012M in 5.016818s Comparison: read varint (pure ruby): 423774.9 i/s read leb128 (native) : 201781.3 i/s - 2.10x slower ``` On the writing path, LEB128 packing outperforms the pure ruby implementation, which is expected. However, on the reading path, LEB128 unpacking does not help the performance due to the "variable length" nature of LEB128, which requires finding exact end of the LEB128 before it can be unpacked: ``` ruby buffer = [] loop do byte = io.readbyte buffer << byte break if byte < 0x80 # this is the last byte of a LEB128 number end buffer.pack('C*').unpack1('R') ``` Because of the extra intermediate steps that locate the ending of LEB128 and build a buffer, using LEB128 unpacking in a streaming context ended up to be much slower than a pure ruby implementation that does both scanning and decoding in a single loop: ``` ruby value = bits = 0 loop do byte = io.readbyte value |= (byte & 0x7f) << bits bits += 7 break if byte < 0x80 end value ``` Therefore, if we can have a native ruby method that does scanning and decoding from `IO` in one shot, it would be a huge improvement. ``` ruby io.unpack1('R') ``` ---Files-------------------------------- benchmark.rb (1.64 KB) -- https://bugs.ruby-lang.org/