[ruby-core:127083] [Ruby Feature#22431] Add `IO#unpack` and `IO#unpack1`
Issue #22431 has been reported by ntkme (Natsuki Natsume). ---------------------------------------- Feature #22431: Add `IO#unpack` and `IO#unpack1` https://bugs.ruby-lang.org/issues/22431 * Author: ntkme (Natsuki Natsume) * Status: Open ---------------------------------------- I would like to have ruby add `IO#unpack` and `IO#unpack1` that effectively does both `IO#read(...)` and `String#unpacking(...)` in one call. This can significantly improve the performance of consuming data in a variable-length encoding format (E.g. LEB128) from a non-peekable `IO` stream (e.g. network/stdio). In a streaming `IO` with variable-length encoding, today it requires the consumer to scan the stream byte-by-byte to precisely finding the ending of a variable-length segment (E.g. LEB128). If we can have `IO#unpack`/`IO#unpack1`, it will be able to precisely consume a variable-length segment by itself, without needing a two-step solution that requires lots of intermediate ruby object allocation. --- Here is a read world example: LEB128 pack/unpack was recently added by: https://bugs.ruby-lang.org/issues/21785 A very common use case for this is LEB128 delimited payload streaming. In real world it can be a LEB128 delimited protobuf message stream. For example: ``` length, payload, length, payload, length, payload... ``` In pseudo code, reading and writing such stream would look like: ```ruby def write(payload, io) io.write([payload.length].pack('R')) io.write(payload) end def read(io) length = io.unpack1('R') # ^ This method does not exist, which makes the real world implementation slower and more complicated. # See attached benchmark.rb for working implementations. io.read(length) end ``` --- See attached `benchmark.rb`, which demonstrates the limitation of native unpacking for LEB128: ``` ruby 4.1.0dev (2026-10-12T17:01:34Z master 98fd04f76c) +YJIT +PRISM [arm64-darwin27] Warming up -------------------------------------- write varint (pure ruby) 12.167k i/100ms write leb128 (native) 15.710k i/100ms Calculating ------------------------------------- write varint (pure ruby) 124.060k (± 1.6%) i/s (8.06 μs/i) - 620.517k in 5.001746s write leb128 (native) 157.036k (± 1.1%) i/s (6.37 μs/i) - 785.500k in 5.002040s Comparison: write leb128 (native) : 157035.9 i/s write varint (pure ruby): 124060.1 i/s - 1.27x slower ruby 4.1.0dev (2026-10-12T17:01:34Z master 98fd04f76c) +YJIT +PRISM [arm64-darwin27] Warming up -------------------------------------- read varint (pure ruby) 42.025k i/100ms read leb128 (native) 20.246k i/100ms Calculating ------------------------------------- read varint (pure ruby) 423.775k (± 0.8%) i/s (2.36 μs/i) - 2.143M in 5.057579s read leb128 (native) 201.781k (± 1.4%) i/s (4.96 μs/i) - 1.012M in 5.016818s Comparison: read varint (pure ruby): 423774.9 i/s read leb128 (native) : 201781.3 i/s - 2.10x slower ``` On the writing path, LEB128 packing outperforms the pure ruby implementation, which is expected. However, on the reading path, LEB128 unpacking does not help the performance due to the "variable length" nature of LEB128, which requires finding exact end of the LEB128 before it can be unpacked: ``` ruby buffer = [] loop do byte = io.readbyte buffer << byte break if byte < 0x80 # this is the last byte of a LEB128 number end buffer.pack('C*').unpack1('R') ``` Because of the extra intermediate steps that locate the ending of LEB128 and build a buffer, using LEB128 unpacking in a streaming context ended up to be much slower than a pure ruby implementation that does both scanning and decoding in a single loop: ``` ruby value = bits = 0 loop do byte = io.readbyte value |= (byte & 0x7f) << bits bits += 7 break if byte < 0x80 end value ``` Therefore, if we can have a native ruby method that does scanning and decoding from `IO` in one shot, it would be a huge improvement. ``` ruby io.unpack1('R') ``` ---Files-------------------------------- benchmark.rb (1.64 KB) -- https://bugs.ruby-lang.org/
Issue #22431 has been updated by mame (Yusuke Endoh). I understand the motivation and have wanted this myself, but there seem to be many things to consider. `unpack` has specifiers like `X` and `@` that do not fit streams. We would also need to decide how to handle specifiers with unbounded length, such as `a*`. Also, the protobuf use case looks risky to me. Reading LEB128 from an untrusted stream without a limit can make us read endlessly and build a huge Integer. Protobuf itself avoids this by imposing its own limit of 10 bytes when reading a varint. But LEB128 itself has no length limit, so it would be unnatural for `unpack` to impose one. So a protobuf implementation would likely need its own limit-aware code anyway, and could not use this method. ---------------------------------------- Feature #22431: Add `IO#unpack` and `IO#unpack1` https://bugs.ruby-lang.org/issues/22431#change-119480 * Author: ntkme (Natsuki Natsume) * Status: Open ---------------------------------------- I would like to have ruby add `IO#unpack` and `IO#unpack1` that effectively does both `IO#read(...)` and `String#unpacking(...)` in one call. This can significantly improve the performance of consuming data in a variable-length encoding format (E.g. LEB128) from a non-peekable `IO` stream (e.g. network/stdio). In a streaming `IO` with variable-length encoding, today it requires the consumer to scan the stream byte-by-byte to precisely finding the ending of a variable-length segment (E.g. LEB128). If we can have `IO#unpack`/`IO#unpack1`, it will be able to precisely consume a variable-length segment by itself, without needing a two-step solution that requires lots of intermediate ruby object allocation. --- Here is a read world example: LEB128 pack/unpack was recently added by: https://bugs.ruby-lang.org/issues/21785 A very common use case for this is LEB128 delimited payload streaming. In real world it can be a LEB128 delimited protobuf message stream. For example: ``` length, payload, length, payload, length, payload... ``` In pseudo code, reading and writing such stream would look like: ```ruby def write(payload, io) io.write([payload.length].pack('R')) io.write(payload) end def read(io) length = io.unpack1('R') # ^ This method does not exist, which makes the real world implementation slower and more complicated. # See attached benchmark.rb for working implementations. io.read(length) end ``` --- See attached `benchmark.rb`, which demonstrates the limitation of native unpacking for LEB128: ``` ruby 4.1.0dev (2026-10-12T17:01:34Z master 98fd04f76c) +YJIT +PRISM [arm64-darwin27] Warming up -------------------------------------- write varint (pure ruby) 12.167k i/100ms write leb128 (native) 15.710k i/100ms Calculating ------------------------------------- write varint (pure ruby) 124.060k (± 1.6%) i/s (8.06 μs/i) - 620.517k in 5.001746s write leb128 (native) 157.036k (± 1.1%) i/s (6.37 μs/i) - 785.500k in 5.002040s Comparison: write leb128 (native) : 157035.9 i/s write varint (pure ruby): 124060.1 i/s - 1.27x slower ruby 4.1.0dev (2026-10-12T17:01:34Z master 98fd04f76c) +YJIT +PRISM [arm64-darwin27] Warming up -------------------------------------- read varint (pure ruby) 42.025k i/100ms read leb128 (native) 20.246k i/100ms Calculating ------------------------------------- read varint (pure ruby) 423.775k (± 0.8%) i/s (2.36 μs/i) - 2.143M in 5.057579s read leb128 (native) 201.781k (± 1.4%) i/s (4.96 μs/i) - 1.012M in 5.016818s Comparison: read varint (pure ruby): 423774.9 i/s read leb128 (native) : 201781.3 i/s - 2.10x slower ``` On the writing path, LEB128 packing outperforms the pure ruby implementation, which is expected. However, on the reading path, LEB128 unpacking does not help the performance due to the "variable length" nature of LEB128, which requires finding exact end of the LEB128 before it can be unpacked: ``` ruby buffer = [] loop do byte = io.readbyte buffer << byte break if byte < 0x80 # this is the last byte of a LEB128 number end buffer.pack('C*').unpack1('R') ``` Because of the extra intermediate steps that locate the ending of LEB128 and build a buffer, using LEB128 unpacking in a streaming context ended up to be much slower than a pure ruby implementation that does both scanning and decoding in a single loop: ``` ruby value = bits = 0 loop do byte = io.readbyte value |= (byte & 0x7f) << bits bits += 7 break if byte < 0x80 end value ``` Therefore, if we can have a native ruby method that does scanning and decoding from `IO` in one shot, it would be a huge improvement. ``` ruby io.unpack1('R') ``` ---Files-------------------------------- benchmark.rb (1.64 KB) -- https://bugs.ruby-lang.org/
Issue #22431 has been updated by ntkme (Natsuki Natsume). The current `String#unpack` has its signature as: ``` String#unpack(template, offset: 0, &block) ``` The concern for unbounded IO read can be potentially solved by having a `maxlen` option: ``` IO#unpack(template, maxlen = nil, &block) ``` I agree that `maxlen` does not make sense for `String#unpack`, however in `IO#unpack` it would be the limit for the internal `read` operation as `IO#unpack` is effectively a method that combines blocking `read` and `unpack`. On the other hand, the `offset` option in `String#unpack` does not make sense for `IO` due to `IO#seek`. As for `X` and `@` in template, it should behave like `IO#seek`, which raises `Errno::ESPIPE` if a `IO` does not support positioning/rewinding. ---------------------------------------- Feature #22431: Add `IO#unpack` and `IO#unpack1` https://bugs.ruby-lang.org/issues/22431#change-119481 * Author: ntkme (Natsuki Natsume) * Status: Open ---------------------------------------- I would like to have ruby add `IO#unpack` and `IO#unpack1` that effectively does both `IO#read(...)` and `String#unpacking(...)` in one call. This can significantly improve the performance of consuming data in a variable-length encoding format (E.g. LEB128) from a non-peekable `IO` stream (e.g. network/stdio). In a streaming `IO` with variable-length encoding, today it requires the consumer to scan the stream byte-by-byte to precisely finding the ending of a variable-length segment (E.g. LEB128). If we can have `IO#unpack`/`IO#unpack1`, it will be able to precisely consume a variable-length segment by itself, without needing a two-step solution that requires lots of intermediate ruby object allocation. --- Here is a read world example: LEB128 pack/unpack was recently added by: https://bugs.ruby-lang.org/issues/21785 A very common use case for this is LEB128 delimited payload streaming. In real world it can be a LEB128 delimited protobuf message stream. For example: ``` length, payload, length, payload, length, payload... ``` In pseudo code, reading and writing such stream would look like: ```ruby def write(payload, io) io.write([payload.length].pack('R')) io.write(payload) end def read(io) length = io.unpack1('R') # ^ This method does not exist, which makes the real world implementation slower and more complicated. # See attached benchmark.rb for working implementations. io.read(length) end ``` --- See attached `benchmark.rb`, which demonstrates the limitation of native unpacking for LEB128: ``` ruby 4.1.0dev (2026-10-12T17:01:34Z master 98fd04f76c) +YJIT +PRISM [arm64-darwin27] Warming up -------------------------------------- write varint (pure ruby) 12.167k i/100ms write leb128 (native) 15.710k i/100ms Calculating ------------------------------------- write varint (pure ruby) 124.060k (± 1.6%) i/s (8.06 μs/i) - 620.517k in 5.001746s write leb128 (native) 157.036k (± 1.1%) i/s (6.37 μs/i) - 785.500k in 5.002040s Comparison: write leb128 (native) : 157035.9 i/s write varint (pure ruby): 124060.1 i/s - 1.27x slower ruby 4.1.0dev (2026-10-12T17:01:34Z master 98fd04f76c) +YJIT +PRISM [arm64-darwin27] Warming up -------------------------------------- read varint (pure ruby) 42.025k i/100ms read leb128 (native) 20.246k i/100ms Calculating ------------------------------------- read varint (pure ruby) 423.775k (± 0.8%) i/s (2.36 μs/i) - 2.143M in 5.057579s read leb128 (native) 201.781k (± 1.4%) i/s (4.96 μs/i) - 1.012M in 5.016818s Comparison: read varint (pure ruby): 423774.9 i/s read leb128 (native) : 201781.3 i/s - 2.10x slower ``` On the writing path, LEB128 packing outperforms the pure ruby implementation, which is expected. However, on the reading path, LEB128 unpacking does not help the performance due to the "variable length" nature of LEB128, which requires finding exact end of the LEB128 before it can be unpacked: ``` ruby buffer = [] loop do byte = io.readbyte buffer << byte break if byte < 0x80 # this is the last byte of a LEB128 number end buffer.pack('C*').unpack1('R') ``` Because of the extra intermediate steps that locate the ending of LEB128 and build a buffer, using LEB128 unpacking in a streaming context ended up to be much slower than a pure ruby implementation that does both scanning and decoding in a single loop: ``` ruby value = bits = 0 loop do byte = io.readbyte value |= (byte & 0x7f) << bits bits += 7 break if byte < 0x80 end value ``` Therefore, if we can have a native ruby method that does scanning and decoding from `IO` in one shot, it would be a huge improvement. ``` ruby io.unpack1('R') ``` ---Files-------------------------------- benchmark.rb (1.64 KB) -- https://bugs.ruby-lang.org/
Issue #22431 has been updated by matz (Yukihiro Matsumoto). I understand the need. Reading length-prefixed messages from a stream is common, and doing it efficiently in Ruby is hard today. But I don't want to bring the whole `unpack` template to IO. As @mame pointed out, `X`, `@` and `*` assume random access to a finite string, and giving them stream semantics makes the same template behave differently on String and IO. We would also have to define what happens at EOF in the middle of a value, with nonblocking IO, and in text mode. For fixed-size data, `io.read(n).unpack` already works. The real problem is variable-length data. Could you narrow the proposal to that? It could be a method that reads one variable-length value, or a lower-level primitive that lets Ruby code do it efficiently. Matz. ---------------------------------------- Feature #22431: Add `IO#unpack` and `IO#unpack1` https://bugs.ruby-lang.org/issues/22431#change-119496 * Author: ntkme (Natsuki Natsume) * Status: Open ---------------------------------------- I would like to have ruby add `IO#unpack` and `IO#unpack1` that effectively does both `IO#read(...)` and `String#unpacking(...)` in one call. This can significantly improve the performance of consuming data in a variable-length encoding format (E.g. LEB128) from a non-peekable `IO` stream (e.g. network/stdio). In a streaming `IO` with variable-length encoding, today it requires the consumer to scan the stream byte-by-byte to precisely finding the ending of a variable-length segment (E.g. LEB128). If we can have `IO#unpack`/`IO#unpack1`, it will be able to precisely consume a variable-length segment by itself, without needing a two-step solution that requires lots of intermediate ruby object allocation. --- Here is a read world example: LEB128 pack/unpack was recently added by: https://bugs.ruby-lang.org/issues/21785 A very common use case for this is LEB128 delimited payload streaming. In real world it can be a LEB128 delimited protobuf message stream. For example: ``` length, payload, length, payload, length, payload... ``` In pseudo code, reading and writing such stream would look like: ```ruby def write(payload, io) io.write([payload.length].pack('R')) io.write(payload) end def read(io) length = io.unpack1('R') # ^ This method does not exist, which makes the real world implementation slower and more complicated. # See attached benchmark.rb for working implementations. io.read(length) end ``` --- See attached `benchmark.rb`, which demonstrates the limitation of native unpacking for LEB128: ``` ruby 4.1.0dev (2026-10-12T17:01:34Z master 98fd04f76c) +YJIT +PRISM [arm64-darwin27] Warming up -------------------------------------- write varint (pure ruby) 12.167k i/100ms write leb128 (native) 15.710k i/100ms Calculating ------------------------------------- write varint (pure ruby) 124.060k (± 1.6%) i/s (8.06 μs/i) - 620.517k in 5.001746s write leb128 (native) 157.036k (± 1.1%) i/s (6.37 μs/i) - 785.500k in 5.002040s Comparison: write leb128 (native) : 157035.9 i/s write varint (pure ruby): 124060.1 i/s - 1.27x slower ruby 4.1.0dev (2026-10-12T17:01:34Z master 98fd04f76c) +YJIT +PRISM [arm64-darwin27] Warming up -------------------------------------- read varint (pure ruby) 42.025k i/100ms read leb128 (native) 20.246k i/100ms Calculating ------------------------------------- read varint (pure ruby) 423.775k (± 0.8%) i/s (2.36 μs/i) - 2.143M in 5.057579s read leb128 (native) 201.781k (± 1.4%) i/s (4.96 μs/i) - 1.012M in 5.016818s Comparison: read varint (pure ruby): 423774.9 i/s read leb128 (native) : 201781.3 i/s - 2.10x slower ``` On the writing path, LEB128 packing outperforms the pure ruby implementation, which is expected. However, on the reading path, LEB128 unpacking does not help the performance due to the "variable length" nature of LEB128, which requires finding exact end of the LEB128 before it can be unpacked: ``` ruby buffer = [] loop do byte = io.readbyte buffer << byte break if byte < 0x80 # this is the last byte of a LEB128 number end buffer.pack('C*').unpack1('R') ``` Because of the extra intermediate steps that locate the ending of LEB128 and build a buffer, using LEB128 unpacking in a streaming context ended up to be much slower than a pure ruby implementation that does both scanning and decoding in a single loop: ``` ruby value = bits = 0 loop do byte = io.readbyte value |= (byte & 0x7f) << bits bits += 7 break if byte < 0x80 end value ``` Therefore, if we can have a native ruby method that does scanning and decoding from `IO` in one shot, it would be a huge improvement. ``` ruby io.unpack1('R') ``` ---Files-------------------------------- benchmark.rb (1.64 KB) -- https://bugs.ruby-lang.org/
Issue #22431 has been updated by ntkme (Natsuki Natsume). Here is a new proposal based on the feedback. For reading variable length data, conceptually it's very similar to the existing `IO#readline(sep, limit, chomp: false) → string)`, except that the `sep` is not a String, but typically a char with a specific range limit. Using a `Proc` to filter every single byte would not be very performant. Meanwhile, we want to make sure it is flexible enough for different kind of variable length data - perhaps we can reuse the existing `IO#readline`, but add the support for `sep` to be a `Range`. For example: ``` ruby range = (..0x80) limit = 10 # limit leb128 for up to 10 bytes for safety with untrusted stream leb128 = io.readline(range, limit, chomp: false) length = leb128.unpack1('R') ``` ---------------------------------------- Feature #22431: Add `IO#unpack` and `IO#unpack1` https://bugs.ruby-lang.org/issues/22431#change-119505 * Author: ntkme (Natsuki Natsume) * Status: Open ---------------------------------------- I would like to have ruby add `IO#unpack` and `IO#unpack1` that effectively does both `IO#read(...)` and `String#unpacking(...)` in one call. This can significantly improve the performance of consuming data in a variable-length encoding format (E.g. LEB128) from a non-peekable `IO` stream (e.g. network/stdio). In a streaming `IO` with variable-length encoding, today it requires the consumer to scan the stream byte-by-byte to precisely finding the ending of a variable-length segment (E.g. LEB128). If we can have `IO#unpack`/`IO#unpack1`, it will be able to precisely consume a variable-length segment by itself, without needing a two-step solution that requires lots of intermediate ruby object allocation. --- Here is a read world example: LEB128 pack/unpack was recently added by: https://bugs.ruby-lang.org/issues/21785 A very common use case for this is LEB128 delimited payload streaming. In real world it can be a LEB128 delimited protobuf message stream. For example: ``` length, payload, length, payload, length, payload... ``` In pseudo code, reading and writing such stream would look like: ```ruby def write(payload, io) io.write([payload.length].pack('R')) io.write(payload) end def read(io) length = io.unpack1('R') # ^ This method does not exist, which makes the real world implementation slower and more complicated. # See attached benchmark.rb for working implementations. io.read(length) end ``` --- See attached `benchmark.rb`, which demonstrates the limitation of native unpacking for LEB128: ``` ruby 4.1.0dev (2026-10-12T17:01:34Z master 98fd04f76c) +YJIT +PRISM [arm64-darwin27] Warming up -------------------------------------- write varint (pure ruby) 12.167k i/100ms write leb128 (native) 15.710k i/100ms Calculating ------------------------------------- write varint (pure ruby) 124.060k (± 1.6%) i/s (8.06 μs/i) - 620.517k in 5.001746s write leb128 (native) 157.036k (± 1.1%) i/s (6.37 μs/i) - 785.500k in 5.002040s Comparison: write leb128 (native) : 157035.9 i/s write varint (pure ruby): 124060.1 i/s - 1.27x slower ruby 4.1.0dev (2026-10-12T17:01:34Z master 98fd04f76c) +YJIT +PRISM [arm64-darwin27] Warming up -------------------------------------- read varint (pure ruby) 42.025k i/100ms read leb128 (native) 20.246k i/100ms Calculating ------------------------------------- read varint (pure ruby) 423.775k (± 0.8%) i/s (2.36 μs/i) - 2.143M in 5.057579s read leb128 (native) 201.781k (± 1.4%) i/s (4.96 μs/i) - 1.012M in 5.016818s Comparison: read varint (pure ruby): 423774.9 i/s read leb128 (native) : 201781.3 i/s - 2.10x slower ``` On the writing path, LEB128 packing outperforms the pure ruby implementation, which is expected. However, on the reading path, LEB128 unpacking does not help the performance due to the "variable length" nature of LEB128, which requires finding exact end of the LEB128 before it can be unpacked: ``` ruby buffer = [] loop do byte = io.readbyte buffer << byte break if byte < 0x80 # this is the last byte of a LEB128 number end buffer.pack('C*').unpack1('R') ``` Because of the extra intermediate steps that locate the ending of LEB128 and build a buffer, using LEB128 unpacking in a streaming context ended up to be much slower than a pure ruby implementation that does both scanning and decoding in a single loop: ``` ruby value = bits = 0 loop do byte = io.readbyte value |= (byte & 0x7f) << bits bits += 7 break if byte < 0x80 end value ``` Therefore, if we can have a native ruby method that does scanning and decoding from `IO` in one shot, it would be a huge improvement. ``` ruby io.unpack1('R') ``` ---Files-------------------------------- benchmark.rb (1.64 KB) -- https://bugs.ruby-lang.org/
participants (3)
-
mame (Yusuke Endoh) -
matz (Yukihiro Matsumoto)
-
ntkme (Natsuki Natsume)