[ruby-core:125953] [Ruby Feature#22185] Add alignment directives to `Array#pack` and `String#unpack`
Issue #22185 has been reported by nobu (Nobuyoshi Nakada). ---------------------------------------- Feature #22185: Add alignment directives to `Array#pack` and `String#unpack` https://bugs.ruby-lang.org/issues/22185 * Author: nobu (Nobuyoshi Nakada) * Status: Open ---------------------------------------- `Array#pack` and `String#unpack` can express byte skips and absolute positions with `x`, `X`, and `@`, but they do not currently have a direct way to align the current position to a byte boundary or to the ABI alignment of another directive. This makes it awkward to describe binary formats that follow C-like structure layout rules. Users need to calculate padding manually, and the required padding can depend on the platform ABI. I propose adding alignment forms to pack templates. ### Alignment Directive ```ruby [1, 2].pack("C x!4 C") # => "\x01\x00\x00\x00\x02" ``` `x!N` advances to the next `N`-byte boundary, filling with NUL bytes when packing and skipping bytes when unpacking. The alignment can also be specified by another directive: ```ruby [1, 2].pack("C x!i i") ``` Here `x!i` aligns to the native alignment of the `i` directive. This lets templates describe platform-dependent native layouts without spelling out the padding size. ### Alignment Mode For formats that need repeated C-structure-style alignment, add an alignment mode: ```ruby [1, 2].pack("!{C i}") ``` This is equivalent to inserting the necessary alignment before each numeric or pointer directive, for example: ```ruby [1, 2].pack("C x!i i") ``` The alignment mode is intentionally not nested. ### Relative and Absolute Alignment There is also a distinction between relative and absolute alignment when `buffer:` or `offset:` is used. `x!` aligns relative to the starting position of the current pack/unpack operation, while `@!` aligns relative to the beginning of the output or input string: ```ruby buffer = +"z" [1, 2].pack("C x!4 C", buffer: buffer) buffer # => "z\x01\x00\x00\x00\x02" buffer = +"z" [1, 2].pack("C @!4 C", buffer: buffer) buffer # => "z\x01\x00\x00\x02" ``` Similarly for unpacking: ```ruby "z\x01\x00\x00\x00\x02".unpack("C x!4 C", offset: 1) # => [1, 2] "z\x01\x00\x00\x02".unpack("C @!4 C", offset: 1) # => [1, 2] ``` This keeps `x!` useful for templates that describe a sub-layout starting at the given `buffer:`/`offset:` position, while `@!` covers formats whose alignment is based on the absolute position in the whole binary string. ### Caveats The directive-alignment form rejects endian modifiers, since byte order does not affect ABI alignment. For example, `x!i` is meaningful, but `x!i<` should not be accepted as an alignment specifier. The alignment is for the pack/unpack offset, not a guarantee about the native address of the underlying String buffer. Callers that need to pass the buffer to C as a typed pointer must ensure separately that the actual address is suitably aligned. Shared substrings, for example, may start at an arbitrary byte offset. ### Examples A concrete use case is building or reading binary records whose layout is shared with C code. For example, a file format, IPC message, device command buffer, or FFI payload may contain records equivalent to: ```C struct entry { uint8_t tag; uint32_t size; void *data; }; ``` Today, a Ruby template has to spell out the padding manually: ```ruby [tag, size, data].pack("C xxx L J") ``` This is both error-prone and ABI-dependent. The padding after `tag` depends on the native alignment of `uint32_t`, and the next field may depend on pointer alignment. With alignment directives, the template can express the intended layout directly: ```ruby [tag, size, data].pack("!{C L J}") ``` or, when only one boundary is needed: ```ruby [tag, size].pack("C x!L L") ``` One target use case is binary buffers that contain several C-like record types. For example, an IPC or device buffer may store a sequence of records, each with its own layout. It is natural to pack or unpack each record with a separate template at a given `buffer:` or `offset:` position. In that case `x!` is useful for alignment relative to the start of the current record, while `@!` is useful for formats whose alignment is defined relative to the start of the whole buffer. -- https://bugs.ruby-lang.org/
Issue #22185 has been updated by matz (Yukihiro Matsumoto). I accept the alignment directives `x!` and `@!`. They fill a real gap: describing C-like structure layouts currently requires manual, ABI-dependent padding calculation, and the directive-alignment form (`x!i`) solves that nicely. I am not yet convinced about the alignment mode `!{...}`. A new grouping syntax in pack templates needs more consideration. Let us start with `x!`/`@!` and revisit the mode later. Matz. ---------------------------------------- Feature #22185: Add alignment directives to `Array#pack` and `String#unpack` https://bugs.ruby-lang.org/issues/22185#change-117996 * Author: nobu (Nobuyoshi Nakada) * Status: Open ---------------------------------------- `Array#pack` and `String#unpack` can express byte skips and absolute positions with `x`, `X`, and `@`, but they do not currently have a direct way to align the current position to a byte boundary or to the ABI alignment of another directive. This makes it awkward to describe binary formats that follow C-like structure layout rules. Users need to calculate padding manually, and the required padding can depend on the platform ABI. I propose adding alignment forms to pack templates. ### Alignment Directive ```ruby [1, 2].pack("C x!4 C") # => "\x01\x00\x00\x00\x02" ``` `x!N` advances to the next `N`-byte boundary, filling with NUL bytes when packing and skipping bytes when unpacking. The alignment can also be specified by another directive: ```ruby [1, 2].pack("C x!i i") ``` Here `x!i` aligns to the native alignment of the `i` directive. This lets templates describe platform-dependent native layouts without spelling out the padding size. ### Alignment Mode For formats that need repeated C-structure-style alignment, add an alignment mode: ```ruby [1, 2].pack("!{C i}") ``` This is equivalent to inserting the necessary alignment before each numeric or pointer directive, for example: ```ruby [1, 2].pack("C x!i i") ``` The alignment mode is intentionally not nested. ### Relative and Absolute Alignment There is also a distinction between relative and absolute alignment when `buffer:` or `offset:` is used. `x!` aligns relative to the starting position of the current pack/unpack operation, while `@!` aligns relative to the beginning of the output or input string: ```ruby buffer = +"z" [1, 2].pack("C x!4 C", buffer: buffer) buffer # => "z\x01\x00\x00\x00\x02" buffer = +"z" [1, 2].pack("C @!4 C", buffer: buffer) buffer # => "z\x01\x00\x00\x02" ``` Similarly for unpacking: ```ruby "z\x01\x00\x00\x00\x02".unpack("C x!4 C", offset: 1) # => [1, 2] "z\x01\x00\x00\x02".unpack("C @!4 C", offset: 1) # => [1, 2] ``` This keeps `x!` useful for templates that describe a sub-layout starting at the given `buffer:`/`offset:` position, while `@!` covers formats whose alignment is based on the absolute position in the whole binary string. ### Caveats The directive-alignment form rejects endian modifiers, since byte order does not affect ABI alignment. For example, `x!i` is meaningful, but `x!i<` should not be accepted as an alignment specifier. The alignment is for the pack/unpack offset, not a guarantee about the native address of the underlying String buffer. Callers that need to pass the buffer to C as a typed pointer must ensure separately that the actual address is suitably aligned. Shared substrings, for example, may start at an arbitrary byte offset. ### Examples A concrete use case is building or reading binary records whose layout is shared with C code. For example, a file format, IPC message, device command buffer, or FFI payload may contain records equivalent to: ```C struct entry { uint8_t tag; uint32_t size; void *data; }; ``` Today, a Ruby template has to spell out the padding manually: ```ruby [tag, size, data].pack("C xxx L J") ``` This is both error-prone and ABI-dependent. The padding after `tag` depends on the native alignment of `uint32_t`, and the next field may depend on pointer alignment. With alignment directives, the template can express the intended layout directly: ```ruby [tag, size, data].pack("!{C L J}") ``` or, when only one boundary is needed: ```ruby [tag, size].pack("C x!L L") ``` One target use case is binary buffers that contain several C-like record types. For example, an IPC or device buffer may store a sequence of records, each with its own layout. It is natural to pack or unpack each record with a separate template at a given `buffer:` or `offset:` position. In that case `x!` is useful for alignment relative to the start of the current record, while `@!` is useful for formats whose alignment is defined relative to the start of the whole buffer. -- https://bugs.ruby-lang.org/
participants (2)
-
matz (Yukihiro Matsumoto) -
nobu (Nobuyoshi Nakada)