Issue #21765 has been updated by YO4 (Yoshinao Muramatsu). I now consider [GH-PR#15408](https://github.com/ruby/ruby/pull/15408) to be a too invasive change. If we can make the encoding converter for CRLF conversion fast enough for practical use, the code that needs to be changed will be sufficiently isolated. However, this means that on Windows, `open(name, "r")` will return a `#<IO>` with encoding conversion enabled, which introduces incompatibility with the behavior of `ungetc`. Regarding this incompatibility, since a test recently added to ruby/spec was related to it, I raised https://github.com/ruby/spec/issues/1387. Regardless, whether to accept the incompatibility and proceed with the change will be discussed and decided within CRuby. Would it be acceptable for `open(name, "r")` on Windows to return a `#<IO>` with encoding conversion enabled? For reference, using `ruby -Eiso-8859-1:utf-8` will result in this incompatibility on other platforms as well. We are already facing this incompatibility. ---------------------------------------- Bug #21765: stop using the C runtime _read() on Windows https://bugs.ruby-lang.org/issues/21765#change-118631 * Author: YO4 (Yoshinao Muramatsu) * Status: Assigned * Assignee: windows * Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN ---------------------------------------- When creating an IO instance in Windows, the default data mode is text mode. In reality, the IO encoding conversion mechanism is not used when encoding conversion is not performed. Instead, the CRLF conversion provided by the C runtime's `_read()` is used. This is explicitly for speed. https://bugs.ruby-lang.org/issues/6401#note-4 As a trade-off, `SET_BINARY_MODE(fptr)` and `SET_BINARY_MODE_WITH_SEEK_CUR(fptr)` are used in various places within io.c, altering the state of the file descriptor. This made the flow of operations difficult to understand and changes hard to implement, especially for developers on other platforms. Additionally, the issues I recently reported were discovered while verifying the impact of modifying the CRLF conversion to utilize the encoding conversion mechanism. #21691 On Windows some of binary read functions of IO are not functional #21687 IOļ¼pos goes wrong after EOF character(ctrl-z) met #21634 Combining read(1) with eof? causes dropout of results unexpectedly on Windows. These issues arise because data read into the rbuf does not match the stream due to newline conversion, or because the buffer end and file position do not align when CTRLZ is detected. As a fix for Bug #21687, I created PR #15216. However, this relies on the internal behavior of the C runtime's `_read()` function, and it seems there is no way to avoid this dependency. **I propose removing the use of C runtime _read().** Reason for Proposal - The mismatch between rbuf and stream contents complicates io_unread() and makes maintenance difficult. - Changing the O_BINARY/O_TEXT state of the file descriptor in various places hinders understanding of the behavior and makes modifications difficult. Two methods to remove C runtime _read() while maintaining current behavior 1. Interpret CRLF and CTRLZ when reading rbuf within io.c. 1. Interpret CRLF and CTRLZ within the encoding conversion framework. My initial idea was to implement the second, using encoding conversion. However, this internally changes the read operation from rbuf to cbuf, resulting in a change to the behavior of ungetc. The proposal in Bug #21682 attempted to generalize this change to minimize its impact. https://bugs.ruby-lang.org/issues/21682 **This issue proposes the first method, crlf conversion during rbuf read.** Problems caused by inconsistencies between the rbuf and stream contents are avoided, and io_unread() becomes the same as on other platforms. Compared to implementing it as an encoding conversion, the advantage is that there is no change in behavior. On the other hand, since each read method in io.c requires individual handling, using encoding conversion results in more localized changes. -- https://bugs.ruby-lang.org/