Issue #22230 has been updated by YO4 (Yoshinao Muramatsu). This may overlap with the discussion in ruby/spec, but I'll share my understanding of the current situation with CRuby. The following illustrates the read model for IO in CRuby's implementation. ``` device -> [rbuf] -> encoding converter -> [cbuf] -> character read -> [ruby string] | + - > byte oriented read -> [ruby string] ``` When reading bytes after reading characters, it is necessary to calculate the current position in rbuf corresponding to the current position in cbuf; however, the reverse conversion from cbuf to rbuf is not straightforward. For example, in the universal newline conversion, \n, \r, and \r\n are converted to \n, so a single \n may correspond to either 1 byte or 2 bytes in rbuf. The current implementation appears to have opted to avoid this reverse conversion. To prevent characters from remaining in cbuf, it performs encoding conversion one character at a time and immediately consumes cbuf during reads that are not whole reads. This is why byte-oriented reads are prohibited when `ungetc` was called, and it is also the cause of the slowness when encoding conversion is enabled. When the encoding converter is not used, the `[rbuf] -> encoding converter` step is omitted, and character operations handle rbuf directly; therefore, there is no difference between character and byte operations. In this case, byte reads are still possible after `ungetc`. The difference in `ungetc` behavior depending on the presence or absence of an encoding converter seems to be a compromise resulting from the fact that the encoding converter was added later while preserving the previous behavior. ---------------------------------------- Misc #22230: Should Ruby specs define byte oriented read behavior when the character buffer is not empty? https://bugs.ruby-lang.org/issues/22230#change-118373 * Author: javanthropus (Jeremy Bopp) * Status: Open ---------------------------------------- In https://github.com/ruby/spec/pull/1384, there was a discussion regarding whether or not raising an IOError when performing a byte oriented read operation while the character buffer is nonempty should be part of the Ruby spec. @eregon asked me to remove the new tests I was requesting to add and also remove existing tests that would define this behavior since it was unclear if the behavior in CRuby was an intentional design decision or a consequence of its specific implementation. I intend to remove the tests in the above MR via a single commit that is easy to revert pending the outcome of this ticket as requested. In all cases I tested, CRuby raises IOError for all byte oriented read operations whenever the character buffer is nonempty. `IO.getc` is used in all these cases to ensure that the character buffer has data in it, but it was argued that other implementations could instead make `getc` reverse the encoding conversion (if one is in play) and put the bytes into the byte buffer instead, eliminating the character buffer entirely. If that conversion would fail, `getc` could raise a conversion error of some kind and leave the buffer unchanged. It's unclear to me though if there are other ways that content could be left in the character buffer during normal read operations which would be harder to avoid while preserving performance and semantics. CRuby's implementation appears to be consistent, so I don't think these tests are attempting to enforce buggy behavior. Should this behavior be codified as part of the spec, or should other implementations be free to behave differently? -- https://bugs.ruby-lang.org/

Issue #22230 has been updated by javanthropus (Jeremy Bopp). Thank you for the responses, @matz and @YO4. That's what I needed to know. ---------------------------------------- Misc #22230: Should Ruby specs define byte oriented read behavior when the character buffer is not empty? https://bugs.ruby-lang.org/issues/22230#change-118449 * Author: javanthropus (Jeremy Bopp) * Status: Open ---------------------------------------- In https://github.com/ruby/spec/pull/1384, there was a discussion regarding whether or not raising an IOError when performing a byte oriented read operation while the character buffer is nonempty should be part of the Ruby spec. @eregon asked me to remove the new tests I was requesting to add and also remove existing tests that would define this behavior since it was unclear if the behavior in CRuby was an intentional design decision or a consequence of its specific implementation. I intend to remove the tests in the above MR via a single commit that is easy to revert pending the outcome of this ticket as requested. In all cases I tested, CRuby raises IOError for all byte oriented read operations whenever the character buffer is nonempty. `IO.getc` is used in all these cases to ensure that the character buffer has data in it, but it was argued that other implementations could instead make `getc` reverse the encoding conversion (if one is in play) and put the bytes into the byte buffer instead, eliminating the character buffer entirely. If that conversion would fail, `getc` could raise a conversion error of some kind and leave the buffer unchanged. It's unclear to me though if there are other ways that content could be left in the character buffer during normal read operations which would be harder to avoid while preserving performance and semantics. CRuby's implementation appears to be consistent, so I don't think these tests are attempting to enforce buggy behavior. Should this behavior be codified as part of the spec, or should other implementations be free to behave differently? -- https://bugs.ruby-lang.org/