[ruby-core:123871] [Ruby Feature#21700] `IO::Buffer.map`: offset argument is "broken" and needs to be made more useful
Issue #21700 has been reported by trinistr (Alexander Bulancov). ---------------------------------------- Feature #21700: `IO::Buffer.map`: offset argument is "broken" and needs to be made more useful https://bugs.ruby-lang.org/issues/21700 * Author: trinistr (Alexander Bulancov) * Status: Open ---------------------------------------- *This was supposed to be a bug report, but it turned into a feature request halfway through.* `IO::Buffer.map` supports `size` and `offset` arguments to localize buffer to a specific part of mapped file. While `size` works fine, specifying `offset` as anything but 0 is a recipe for disaster. 1. It is practically undocumented. This is it currently: "Optional size and offset of mapping can be specified." While `size` can be whatever user desires, `offset` can only be set to specific values. It was *very* surprising when I started testing it, as I expected it to behave similar to size and offset of a slice. Which values are allowed? That's the next point. 2. Possible values for `offset` are highly platform-dependent. - On Linux it must be a multiple of page size. (https://www.man7.org/linux/man-pages/man2/mmap.2.html) - On macOS it can be any value? manpage doesn't specify exactly. (https://developer.apple.com/library/archive/documentation/System/Conceptual/... (I looked at the manpage on a mac, seems to be the same.)) - On Windows it must be a multiple of some value that *isn't* page size and page size doesn't work here. StackOverflow says it's 64 kB, but that didn't work for me. (See dwFileOffsetLow in https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-m...) - On other systems? Who knows. 3. On macOS, a buffer successfully created with an offset crashes on read. I don't know why, it's exactly the same file, size and offset that works on Ubuntu, and the read is bounded. 4. Currently, offset doesn't change the buffer's size if unspecified, leading to crashes. I will be making a PR addressing this but still. 5. All of this makes offset unpractical to test, leading to all kinds of problems. So the problem is: this argument is a very low-level detail that requires significant knowledge of all platforms your code may be ever deployed to, even if strictly Ruby is used (no C API), with unexpectedly high footgun potential. I understand that IO::Buffer is a lower-level class, but this seems too much. I propose to make `offset` argument into a convenience for users, allowing any reasonable value (i.e inside the file). Behind the scenes, we can use the native offset for efficieny, on platforms where it works well, but the user API would be decoupled from this. However, I'm not particularly familiar with Ruby internals, so I can't tell how viable this is. -- https://bugs.ruby-lang.org/
Issue #21700 has been updated by trinistr (Alexander Bulancov).
Currently, offset doesn't change the buffer's size if unspecified, leading to crashes. I will be making a PR addressing this but still.
The pull request in question: https://github.com/ruby/ruby/pull/15264 ---------------------------------------- Feature #21700: `IO::Buffer.map`: offset argument is "broken" and needs to be made more useful https://bugs.ruby-lang.org/issues/21700#change-115274 * Author: trinistr (Alexander Bulancov) * Status: Open ---------------------------------------- *This was supposed to be a bug report, but it turned into a feature request halfway through.* `IO::Buffer.map` supports `size` and `offset` arguments to localize buffer to a specific part of mapped file. While `size` works fine, specifying `offset` as anything but 0 is a recipe for disaster. 1. It is practically undocumented. This is it currently: "Optional size and offset of mapping can be specified." While `size` can be whatever user desires, `offset` can only be set to specific values. It was *very* surprising when I started testing it, as I expected it to behave similar to size and offset of a slice. Which values are allowed? That's the next point. 2. Possible values for `offset` are highly platform-dependent. - On Linux it must be a multiple of page size. (https://www.man7.org/linux/man-pages/man2/mmap.2.html) - On macOS it can be any value? manpage doesn't specify exactly. (https://developer.apple.com/library/archive/documentation/System/Conceptual/... (I looked at the manpage on a mac, seems to be the same.)) - On Windows it must be a multiple of some value that *isn't* page size and page size doesn't work here. StackOverflow says it's 64 kB, but that didn't work for me. (See dwFileOffsetLow in https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-m...) - On other systems? Who knows. 3. On macOS, a buffer successfully created with an offset crashes on read. I don't know why, it's exactly the same file, size and offset that works on Ubuntu, and the read is bounded. 4. Currently, offset doesn't change the buffer's size if unspecified, leading to crashes. I will be making a PR addressing this but still. 5. All of this makes offset unpractical to test, leading to all kinds of problems. So the problem is: this argument is a very low-level detail that requires significant knowledge of all platforms your code may be ever deployed to, even if strictly Ruby is used (no C API), with unexpectedly high footgun potential. I understand that IO::Buffer is a lower-level class, but this seems too much. I propose to make `offset` argument into a convenience for users, allowing any reasonable value (i.e inside the file). Behind the scenes, we can use the native offset for efficieny, on platforms where it works well, but the user API would be decoupled from this. However, I'm not particularly familiar with Ruby internals, so I can't tell how viable this is. -- https://bugs.ruby-lang.org/
Issue #21700 has been updated by ioquatix (Samuel Williams). This makes sense to me. However it might mean we need to return a slice (which IMHO is probably okay). The alternative is to expose the internal implementation and force the user to deal with all that shit. We should also document clearly that if offset is non-zero, that a slice may be returned on some platforms, depending on what the underlying system supports. ---------------------------------------- Feature #21700: `IO::Buffer.map`: offset argument is "broken" and needs to be made more useful https://bugs.ruby-lang.org/issues/21700#change-115275 * Author: trinistr (Alexander Bulancov) * Status: Open ---------------------------------------- *This was supposed to be a bug report, but it turned into a feature request halfway through.* `IO::Buffer.map` supports `size` and `offset` arguments to localize buffer to a specific part of mapped file. While `size` works fine, specifying `offset` as anything but 0 is a recipe for disaster. 1. It is practically undocumented. This is it currently: "Optional size and offset of mapping can be specified." While `size` can be whatever user desires, `offset` can only be set to specific values. It was *very* surprising when I started testing it, as I expected it to behave similar to size and offset of a slice. Which values are allowed? That's the next point. 2. Possible values for `offset` are highly platform-dependent. - On Linux it must be a multiple of page size. (https://www.man7.org/linux/man-pages/man2/mmap.2.html) - On macOS it can be any value? manpage doesn't specify exactly. (https://developer.apple.com/library/archive/documentation/System/Conceptual/... (I looked at the manpage on a mac, seems to be the same.)) - On Windows it must be a multiple of some value that *isn't* page size and page size doesn't work here. StackOverflow says it's 64 kB, but that didn't work for me. (See dwFileOffsetLow in https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-m...) - On other systems? Who knows. 3. On macOS, a buffer successfully created with an offset crashes on read. I don't know why, it's exactly the same file, size and offset that works on Ubuntu, and the read is bounded. 4. Currently, offset doesn't change the buffer's size if unspecified, leading to crashes. I will be making a PR addressing this but still. 5. All of this makes offset unpractical to test, leading to all kinds of problems. So the problem is: this argument is a very low-level detail that requires significant knowledge of all platforms your code may be ever deployed to, even if strictly Ruby is used (no C API), with unexpectedly high footgun potential. I understand that IO::Buffer is a lower-level class, but this seems too much. I propose to make `offset` argument into a convenience for users, allowing any reasonable value (i.e inside the file). Behind the scenes, we can use the native offset for efficieny, on platforms where it works well, but the user API would be decoupled from this. However, I'm not particularly familiar with Ruby internals, so I can't tell how viable this is. -- https://bugs.ruby-lang.org/
Issue #21700 has been updated by trinistr (Alexander Bulancov). I think slices always require a source buffer? If so, returning a slice will require somehow handling a buffer so it doesn't leak (though GC may take care of it?). On a related note, slices require some work on their own, there is no `#slice?` method for example and so on. My gripes with them probably need a separate feature issue. ---------------------------------------- Feature #21700: `IO::Buffer.map`: offset argument is "broken" and needs to be made more useful https://bugs.ruby-lang.org/issues/21700#change-115281 * Author: trinistr (Alexander Bulancov) * Status: Open ---------------------------------------- *This was supposed to be a bug report, but it turned into a feature request halfway through.* `IO::Buffer.map` supports `size` and `offset` arguments to localize buffer to a specific part of mapped file. While `size` works fine, specifying `offset` as anything but 0 is a recipe for disaster. 1. It is practically undocumented. This is it currently: "Optional size and offset of mapping can be specified." While `size` can be whatever user desires, `offset` can only be set to specific values. It was *very* surprising when I started testing it, as I expected it to behave similar to size and offset of a slice. Which values are allowed? That's the next point. 2. Possible values for `offset` are highly platform-dependent. - On Linux it must be a multiple of page size. (https://www.man7.org/linux/man-pages/man2/mmap.2.html) - On macOS it can be any value? manpage doesn't specify exactly. (https://developer.apple.com/library/archive/documentation/System/Conceptual/... (I looked at the manpage on a mac, seems to be the same.)) - On Windows it must be a multiple of some value that *isn't* page size and page size doesn't work here. StackOverflow says it's 64 kB, but that didn't work for me. (See dwFileOffsetLow in https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-m...) - On other systems? Who knows. 3. On macOS, a buffer successfully created with an offset crashes on read. I don't know why, it's exactly the same file, size and offset that works on Ubuntu, and the read is bounded. 4. Currently, offset doesn't change the buffer's size if unspecified, leading to crashes. I will be making a PR addressing this but still. 5. All of this makes offset unpractical to test, leading to all kinds of problems. So the problem is: this argument is a very low-level detail that requires significant knowledge of all platforms your code may be ever deployed to, even if strictly Ruby is used (no C API), with unexpectedly high footgun potential. I understand that IO::Buffer is a lower-level class, but this seems too much. I propose to make `offset` argument into a convenience for users, allowing any reasonable value (i.e inside the file). Behind the scenes, we can use the native offset for efficieny, on platforms where it works well, but the user API would be decoupled from this. However, I'm not particularly familiar with Ruby internals, so I can't tell how viable this is. -- https://bugs.ruby-lang.org/
Issue #21700 has been updated by ioquatix (Samuel Williams). I've been using `IO::Buffer.map` on macOS without issues. It's still early days but I wonder why it was failing for you. https://github.com/socketry/async-utilization/blob/main/lib/async/utilizatio... ---------------------------------------- Feature #21700: `IO::Buffer.map`: offset argument is "broken" and needs to be made more useful https://bugs.ruby-lang.org/issues/21700#change-116683 * Author: trinistr (Alexander Bulancov) * Status: Open ---------------------------------------- *This was supposed to be a bug report, but it turned into a feature request halfway through.* `IO::Buffer.map` supports `size` and `offset` arguments to localize buffer to a specific part of mapped file. While `size` works fine, specifying `offset` as anything but 0 is a recipe for disaster. 1. It is practically undocumented. This is it currently: "Optional size and offset of mapping can be specified." While `size` can be whatever user desires, `offset` can only be set to specific values. It was *very* surprising when I started testing it, as I expected it to behave similar to size and offset of a slice. Which values are allowed? That's the next point. 2. Possible values for `offset` are highly platform-dependent. - On Linux it must be a multiple of page size. (https://www.man7.org/linux/man-pages/man2/mmap.2.html) - On macOS it can be any value? manpage doesn't specify exactly. (https://developer.apple.com/library/archive/documentation/System/Conceptual/... (I looked at the manpage on a mac, seems to be the same.)) - On Windows it must be a multiple of some value that *isn't* page size and page size doesn't work here. StackOverflow says it's 64 kB, but that didn't work for me. (See dwFileOffsetLow in https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-m...) - On other systems? Who knows. 3. On macOS, a buffer successfully created with an offset crashes on read. I don't know why, it's exactly the same file, size and offset that works on Ubuntu, and the read is bounded. 4. Currently, offset doesn't change the buffer's size if unspecified, leading to crashes. I will be making a PR addressing this but still. 5. All of this makes offset unpractical to test, leading to all kinds of problems. So the problem is: this argument is a very low-level detail that requires significant knowledge of all platforms your code may be ever deployed to, even if strictly Ruby is used (no C API), with unexpectedly high footgun potential. I understand that IO::Buffer is a lower-level class, but this seems too much. I propose to make `offset` argument into a convenience for users, allowing any reasonable value (i.e inside the file). Behind the scenes, we can use the native offset for efficieny, on platforms where it works well, but the user API would be decoupled from this. However, I'm not particularly familiar with Ruby internals, so I can't tell how viable this is. -- https://bugs.ruby-lang.org/
participants (2)
-
ioquatix (Samuel Williams) -
trinistr (Alexander Bulancov)