Issue #20663 has been updated by javanthropus (Jeremy Bopp). After reviewing the sources, I see that this behavior is a consequence of how character conversion is handled. Fetching the next character always looks at an internal buffer for bytes to compose the character. When that buffer is empty, bytes are fetched from the stream, converted from the external encoding to the internal encoding, and then stored in the buffer. As far as IO#each_char is concerned, the bytes pushed into the buffer via IO#ungetc may as well have been left from the previous buffer fill operation, so there's no way to tell that the next bytes in the buffer should be handled in any special way to assist with character reading recovery from the underlying stream. In short, please close this issue as a non-bug. ---------------------------------------- Bug #20663: Reading characters from IO does not recover gracefully from bad data pushed via IO#ungetc https://bugs.ruby-lang.org/issues/20663#change-109384 * Author: javanthropus (Jeremy Bopp) * Status: Open * ruby -v: ruby 3.3.4 (2024-07-09 revision be1089c8ec) [x86_64-linux] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- If bytes that result in at least 2 invalid characters for the internal encoding of an IO object are pushed into the internal buffer with IO#getc, reading from the stream returns invalid characters composed of both bytes from the internal buffer and the converted bytes from the stream even if the next character in the stream itself is completely valid. ``` ruby char_bytes = Tempfile.open(encoding: 'utf-8:utf-16le') do |f| f.write("🍣") f.rewind f.ungetc("🍣".encode('utf-16le').b[0..-2]) f.each_char.map(&:bytes) end puts char_bytes.inspect ``` The above outputs: ``` [[60, 216], [99, 60], [216, 99], [223]] ``` I expect it to output: ``` [[60, 216], [99], [60, 216, 99, 223]] ``` In other words, I expect it to first completely drain the internal character buffer returning as many characters as necessary (invalid or otherwise) before reading from the stream and converting and returning the next character. After a bit of testing, the way it seems to behave is this: 1. Return the next character from the internal buffer either if it's a valid encoding or if there is more than 1 character in the buffer, valid or not 2. Otherwise, read another character from the stream, convert it, and append the converted bytes to the buffer 3. Go back to step 1 Maybe this is desired behavior, but I can't understand why. It can't recover from the kind of erroneous data demonstrated in the example above. -- https://bugs.ruby-lang.org/