[ruby-core:127331] [Ruby Misc#22506] Clarify read-only, lock and freeze in IO::Buffer
Issue #22506 has been reported by himura467 (Akito Shitara). ---------------------------------------- Misc #22506: Clarify read-only, lock and freeze in IO::Buffer https://bugs.ruby-lang.org/issues/22506 * Author: himura467 (Akito Shitara) * Status: Open ---------------------------------------- Following https://bugs.ruby-lang.org/issues/22274#note-16, I'd like to agree on what each of these means for `IO::Buffer`, so that the remaining fixes follow one model. This is a proposal, and I'd appreciate corrections. | | What it prevents | Duration | |---|---|---| | read-only | writes through this buffer | set at creation | | lock | freeing, resizing or transferring the memory | while a native operation borrows it | | freeze | any change made through this buffer, including writes to its bytes | permanent | None of them promises that the bytes cannot change through another view or another process. Proposed rules: * A frozen buffer refuses writes (`FrozenError`), and a read-only one refuses writes (`AccessError`), like a frozen `File` and a `File` opened with "r". * `readonly?` and the `READONLY` flag from `rb_io_buffer_get_bytes` report whether the memory may be written through this buffer, not whether the buffer is frozen. A frozen source makes its slices read-only, but freezing a buffer does not make the buffer itself read-only, so C code should write through `rb_io_buffer_get_bytes_for_writing`, which checks both. * A frozen buffer cannot be freed, resized or transferred, so locking it is unnecessary and can be a no-op. * A locked buffer cannot be frozen (https://bugs.ruby-lang.org/issues/22274#note-10), like a temporarily locked String. Open questions: 1. The owner of borrowed memory (a buffer from `rb_io_buffer_new` without a source) has to invalidate the buffer before reclaiming the memory, even if the code it was lent to has frozen or locked it. Should invalidation by the owner be an exception to the rules above, or should such a buffer refuse to be frozen? 2. With this model, a frozen buffer is no longer writable. If a buffer that stays writable while its memory is guaranteed to stay accessible is needed beyond what holding a lock provides, I wouldn't object to expressing that state with a separate name such as `#pin` (https://bugs.ruby-lang.org/issues/22274#note-18). @ioquatix, is there a reason or use case for expressing it with `freeze` instead? -- https://bugs.ruby-lang.org/
Issue #22506 has been updated by Eregon (Benoit Daloze). +1, I think this is the intuitive and consistent semantics with the rest of Ruby. himura467 (Akito Shitara) wrote:
A frozen buffer cannot be freed, resized or transferred, so locking it is unnecessary and can be a no-op.
A frozen buffer might still be moved (by the GC) though, which needs to be prevented during native calls (e.g. `write(2)`). That might already work though through the fact that `VALUE`s on the stack are pinned while on the stack. ---------------------------------------- Misc #22506: Clarify read-only, lock and freeze in IO::Buffer https://bugs.ruby-lang.org/issues/22506#change-119766 * Author: himura467 (Akito Shitara) * Status: Open ---------------------------------------- Following https://bugs.ruby-lang.org/issues/22274#note-16, I'd like to agree on what each of these means for `IO::Buffer`, so that the remaining fixes follow one model. This is a proposal, and I'd appreciate corrections. | | What it prevents | Duration | |---|---|---| | read-only | writes through this buffer | set at creation | | lock | freeing, resizing or transferring the memory | while a native operation borrows it | | freeze | any change made through this buffer, including writes to its bytes | permanent | None of them promises that the bytes cannot change through another view or another process. Proposed rules: * A frozen buffer refuses writes (`FrozenError`), and a read-only one refuses writes (`AccessError`), like a frozen `File` and a `File` opened with "r". * `readonly?` and the `READONLY` flag from `rb_io_buffer_get_bytes` report whether the memory may be written through this buffer, not whether the buffer is frozen. A frozen source makes its slices read-only, but freezing a buffer does not make the buffer itself read-only, so C code should write through `rb_io_buffer_get_bytes_for_writing`, which checks both. * A frozen buffer cannot be freed, resized or transferred, so locking it is unnecessary and can be a no-op. * A locked buffer cannot be frozen (https://bugs.ruby-lang.org/issues/22274#note-10), like a temporarily locked String. Open questions: 1. The owner of borrowed memory (a buffer from `rb_io_buffer_new` without a source) has to invalidate the buffer before reclaiming the memory, even if the code it was lent to has frozen or locked it. Should invalidation by the owner be an exception to the rules above, or should such a buffer refuse to be frozen? 2. With this model, a frozen buffer is no longer writable. If a buffer that stays writable while its memory is guaranteed to stay accessible is needed beyond what holding a lock provides, I wouldn't object to expressing that state with a separate name such as `#pin` (https://bugs.ruby-lang.org/issues/22274#note-18). @ioquatix, is there a reason or use case for expressing it with `freeze` instead? -- https://bugs.ruby-lang.org/
Issue #22506 has been updated by himura467 (Akito Shitara). Eregon (Benoit Daloze) wrote in #note-1:
A frozen buffer might still be moved (by the GC) though, which needs to be prevented during native calls (e.g. `write(2)`). That might already work though through the fact that `VALUE`s on the stack are pinned while on the stack.
Thanks. As far as I understand, INTERNAL and MAPPED memory, and external memory passed in from C, are outside the GC heap and never move. The only GC-managed memory a buffer can point into is a String source, which `dmark` pins with `rb_gc_mark`. The `IO::Buffer` object itself could move, but as you said, it stays pinned while it is on the stack during native calls such as `write(2)`. Because of this, `IO::Buffer`'s lock doesn't pin anything either, so I think locking a frozen buffer can be a no-op. ---------------------------------------- Misc #22506: Clarify read-only, lock and freeze in IO::Buffer https://bugs.ruby-lang.org/issues/22506#change-119810 * Author: himura467 (Akito Shitara) * Status: Open ---------------------------------------- Following https://bugs.ruby-lang.org/issues/22274#note-16, I'd like to agree on what each of these means for `IO::Buffer`, so that the remaining fixes follow one model. This is a proposal, and I'd appreciate corrections. | | What it prevents | Duration | |---|---|---| | read-only | writes through this buffer | set at creation | | lock | freeing, resizing or transferring the memory | while a native operation borrows it | | freeze | any change made through this buffer, including writes to its bytes | permanent | None of them promises that the bytes cannot change through another view or another process. Proposed rules: * A frozen buffer refuses writes (`FrozenError`), and a read-only one refuses writes (`AccessError`), like a frozen `File` and a `File` opened with "r". * `readonly?` and the `READONLY` flag from `rb_io_buffer_get_bytes` report whether the memory may be written through this buffer, not whether the buffer is frozen. A frozen source makes its slices read-only, but freezing a buffer does not make the buffer itself read-only, so C code should write through `rb_io_buffer_get_bytes_for_writing`, which checks both. * A frozen buffer cannot be freed, resized or transferred, so locking it is unnecessary and can be a no-op. * A locked buffer cannot be frozen (https://bugs.ruby-lang.org/issues/22274#note-10), like a temporarily locked String. Open questions: 1. The owner of borrowed memory (a buffer from `rb_io_buffer_new` without a source) has to invalidate the buffer before reclaiming the memory, even if the code it was lent to has frozen or locked it. Should invalidation by the owner be an exception to the rules above, or should such a buffer refuse to be frozen? 2. With this model, a frozen buffer is no longer writable. If a buffer that stays writable while its memory is guaranteed to stay accessible is needed beyond what holding a lock provides, I wouldn't object to expressing that state with a separate name such as `#pin` (https://bugs.ruby-lang.org/issues/22274#note-18). @ioquatix, is there a reason or use case for expressing it with `freeze` instead? -- https://bugs.ruby-lang.org/
Issue #22506 has been updated by kou (Kouhei Sutou). Open question 1: This is about the `IO::Buffer.for(str) {|buffer| buffer.freeze}` case, right? I think it should behave the same as `File.open(path) {|file| file.freeze}`: * `freeze` is allowed inside the block * The block-end can free/close the buffer/file * Using the buffer/file outside the block isn't supported Open question 2: @ioquatix It seems that sharing a writable buffer with Ractors in #22274-6 is a use case of "this buffer's address and size is stable" (and this buffer is writable) in #22274-10. Is that correct? If so, I think that we should reject the use case based on the Ractor concept. (It's already mentioned in #22274-7.) I think that we don't need to introduce the "pin" concept because the current "lock" concept is "this buffer's address and size is stable" (and this buffer may be writable). ---------------------------------------- Misc #22506: Clarify read-only, lock and freeze in IO::Buffer https://bugs.ruby-lang.org/issues/22506#change-119838 * Author: himura467 (Akito Shitara) * Status: Open ---------------------------------------- Following https://bugs.ruby-lang.org/issues/22274#note-16, I'd like to agree on what each of these means for `IO::Buffer`, so that the remaining fixes follow one model. This is a proposal, and I'd appreciate corrections. | | What it prevents | Duration | |---|---|---| | read-only | writes through this buffer | set at creation | | lock | freeing, resizing or transferring the memory | while a native operation borrows it | | freeze | any change made through this buffer, including writes to its bytes | permanent | None of them promises that the bytes cannot change through another view or another process. Proposed rules: * A frozen buffer refuses writes (`FrozenError`), and a read-only one refuses writes (`AccessError`), like a frozen `File` and a `File` opened with "r". * `readonly?` and the `READONLY` flag from `rb_io_buffer_get_bytes` report whether the memory may be written through this buffer, not whether the buffer is frozen. A frozen source makes its slices read-only, but freezing a buffer does not make the buffer itself read-only, so C code should write through `rb_io_buffer_get_bytes_for_writing`, which checks both. * A frozen buffer cannot be freed, resized or transferred, so locking it is unnecessary and can be a no-op. * A locked buffer cannot be frozen (https://bugs.ruby-lang.org/issues/22274#note-10), like a temporarily locked String. Open questions: 1. The owner of borrowed memory (a buffer from `rb_io_buffer_new` without a source) has to invalidate the buffer before reclaiming the memory, even if the code it was lent to has frozen or locked it. Should invalidation by the owner be an exception to the rules above, or should such a buffer refuse to be frozen? 2. With this model, a frozen buffer is no longer writable. If a buffer that stays writable while its memory is guaranteed to stay accessible is needed beyond what holding a lock provides, I wouldn't object to expressing that state with a separate name such as `#pin` (https://bugs.ruby-lang.org/issues/22274#note-18). @ioquatix, is there a reason or use case for expressing it with `freeze` instead? -- https://bugs.ruby-lang.org/
Issue #22506 has been updated by himura467 (Akito Shitara). kou (Kouhei Sutou) wrote in #note-3:
Open question 1:
This is about the `IO::Buffer.for(str) {|buffer| buffer.freeze}` case, right? I think it should behave the same as `File.open(path) {|file| file.freeze}`:
* `freeze` is allowed inside the block * The block-end can free/close the buffer/file * Using the buffer/file outside the block isn't supported
Thanks. I wrote question 1 with `rb_io_buffer_new` in mind, where the C code that creates the buffer owns the memory and invalidates the buffer before releasing it. The same applies to `IO::Buffer.for`, so I agree that it should behave like `File.open(path) {|file| file.freeze}`. ---------------------------------------- Misc #22506: Clarify read-only, lock and freeze in IO::Buffer https://bugs.ruby-lang.org/issues/22506#change-119841 * Author: himura467 (Akito Shitara) * Status: Open ---------------------------------------- Following https://bugs.ruby-lang.org/issues/22274#note-16, I'd like to agree on what each of these means for `IO::Buffer`, so that the remaining fixes follow one model. This is a proposal, and I'd appreciate corrections. | | What it prevents | Duration | |---|---|---| | read-only | writes through this buffer | set at creation | | lock | freeing, resizing or transferring the memory | while a native operation borrows it | | freeze | any change made through this buffer, including writes to its bytes | permanent | None of them promises that the bytes cannot change through another view or another process. Proposed rules: * A frozen buffer refuses writes (`FrozenError`), and a read-only one refuses writes (`AccessError`), like a frozen `File` and a `File` opened with "r". * `readonly?` and the `READONLY` flag from `rb_io_buffer_get_bytes` report whether the memory may be written through this buffer, not whether the buffer is frozen. A frozen source makes its slices read-only, but freezing a buffer does not make the buffer itself read-only, so C code should write through `rb_io_buffer_get_bytes_for_writing`, which checks both. * A frozen buffer cannot be freed, resized or transferred, so locking it is unnecessary and can be a no-op. * A locked buffer cannot be frozen (https://bugs.ruby-lang.org/issues/22274#note-10), like a temporarily locked String. Open questions: 1. The owner of borrowed memory (a buffer from `rb_io_buffer_new` without a source) has to invalidate the buffer before reclaiming the memory, even if the code it was lent to has frozen or locked it. Should invalidation by the owner be an exception to the rules above, or should such a buffer refuse to be frozen? 2. With this model, a frozen buffer is no longer writable. If a buffer that stays writable while its memory is guaranteed to stay accessible is needed beyond what holding a lock provides, I wouldn't object to expressing that state with a separate name such as `#pin` (https://bugs.ruby-lang.org/issues/22274#note-18). @ioquatix, is there a reason or use case for expressing it with `freeze` instead? -- https://bugs.ruby-lang.org/
Issue #22506 has been updated by matz (Yukihiro Matsumoto). I agree with the proposed model. But before going further, I'd like to hear the opinion from @ioquatix first. Matz. ---------------------------------------- Misc #22506: Clarify read-only, lock and freeze in IO::Buffer https://bugs.ruby-lang.org/issues/22506#change-119946 * Author: himura467 (Akito Shitara) * Status: Open ---------------------------------------- Following https://bugs.ruby-lang.org/issues/22274#note-16, I'd like to agree on what each of these means for `IO::Buffer`, so that the remaining fixes follow one model. This is a proposal, and I'd appreciate corrections. | | What it prevents | Duration | |---|---|---| | read-only | writes through this buffer | set at creation | | lock | freeing, resizing or transferring the memory | while a native operation borrows it | | freeze | any change made through this buffer, including writes to its bytes | permanent | None of them promises that the bytes cannot change through another view or another process. Proposed rules: * A frozen buffer refuses writes (`FrozenError`), and a read-only one refuses writes (`AccessError`), like a frozen `File` and a `File` opened with "r". * `readonly?` and the `READONLY` flag from `rb_io_buffer_get_bytes` report whether the memory may be written through this buffer, not whether the buffer is frozen. A frozen source makes its slices read-only, but freezing a buffer does not make the buffer itself read-only, so C code should write through `rb_io_buffer_get_bytes_for_writing`, which checks both. * A frozen buffer cannot be freed, resized or transferred, so locking it is unnecessary and can be a no-op. * A locked buffer cannot be frozen (https://bugs.ruby-lang.org/issues/22274#note-10), like a temporarily locked String. Open questions: 1. The owner of borrowed memory (a buffer from `rb_io_buffer_new` without a source) has to invalidate the buffer before reclaiming the memory, even if the code it was lent to has frozen or locked it. Should invalidation by the owner be an exception to the rules above, or should such a buffer refuse to be frozen? 2. With this model, a frozen buffer is no longer writable. If a buffer that stays writable while its memory is guaranteed to stay accessible is needed beyond what holding a lock provides, I wouldn't object to expressing that state with a separate name such as `#pin` (https://bugs.ruby-lang.org/issues/22274#note-18). @ioquatix, is there a reason or use case for expressing it with `freeze` instead? -- https://bugs.ruby-lang.org/
participants (4)
-
Eregon (Benoit Daloze) -
himura467 (Akito Shitara) -
kou (Kouhei Sutou) -
matz (Yukihiro Matsumoto)