Issue #19057 has been updated by kjtsanaktsidis (KJ Tsanaktsidis). I did a bit of research on this topic this evening. Firstly, some technical notes r.e. undefined behaviour. My understanding is that "Current proposal" is really undefined behaviour. This is in the C standard, section 6.5.2.3, point 6 (http://www.open-std.org/jtc1/sc22/wg14/www/abq/c17_updated_proposed_fdis.pdf)

One special guarantee is made in order to simplify the use of unions: if a union contains several structures that share a common initial sequence (see below), and if the union object currently contains one of these structures, it is permitted to inspect the common initial part of any of them anywhere that a declaration of the completed type of the union is visible. Two structures share a common initial sequence if corresponding members have compatible types (and, for bit-fields, the same widths) for a sequence of one or more initial members

It seems that this "common initial sequence" rule is really only for structures accessed through unions. Another fun gotcha is that, even if you _do_ do all the accesses through a union, the compiler can assume that `rb_io_public_t *` and `rb_io_private_t *` should never alias each other (which is the exact opposite of what you want). E.g. - this program has a different result depending on whether optimizations are on or not - http://coliru.stacked-crooked.com/a/b57c8dd9e2ef3a02 I think to be technically correct, we would need to go for the "Nested public interface" approach - structs _can_ be converted to a pointer to their first field according to the C standard (although I guess the _reverse_ cast from `rb_io_public_t *` back to `rb_io_private_t *` would be UB?) ------------ I don't think this matters though, because I agree that, looking at the contents of `rb_io_t`, almost none of this should be public API, and we should strive to encapsulate it fully, like "Hide all details" (or like @Eregon suggested too). I did a Github code search for usages of `rb_io_t`, and _all_ of the usages of it I could find basically fell into this pattern. ``` VALUE some_io = /* from somewhere */; rb_io_t *fptr; GetOpenFile(some_io, fptr) // call an io.h method that takes rb_io_t * rb_io_set_nonblock(fptr); // or read the fd member and do something with that do_something_external_with_fd(fptr->fd); ``` I think we could avoid 99% of the breakage, and still hide all the implementation details of rb_io_t, by basically redefining `rb_io_t` to contain just the FD, and define a new internal, opaque type for internal IO use. We would do this like so: Firstly, `struct RFile` is currently in include/ruby/internal/core/rfile.h as: ``` struct rb_io_t; struct RFile { struct RBasic basic; struct rb_io_t *fptr; }; ``` There are two spare words in there, and we would use one to add a pointer to a new structure. The definition of that structure would _not_ be provided in public headers anyhere. So, we would change RFile in include/ruby/internal/core/rfile.h to be: ``` struct rb_io_t; struct rb_io_impl; struct RFile { struct RBasic basic; struct rb_io_t *fptr; struct rb_io_impl *impl; }; ``` Then, we would change the definition of `struct rb_io_t` in include/ruby/io.h to be: ``` typedef struct rb_io_t { VALUE self; int fd; // _everything_ else is removed } rb_io_t; ``` We would move all of its current contents to a new struct in internal/io.h: ``` struct rb_io_impl { // All the juicy implementation details go here - _except_ no need to duplicate `self` and `fd`. }; ``` The current definition of `GetOpenFile` essentially does `RFILE(obj)->fptr`, so that would return the newly-slimmed-down public `rb_io_t`. Attempts to read the file descriptor off that would continue to work. Other methods in include/ruby/io.h which take a `rb_io_t *` as an argument, like e.g. `rb_io_set_nonblock`, would do `RFILE(fptr->self)->impl` to get access to the implementation details struct. We could also define variants of these which took the `VALUE` instead of the `rb_io_t *` to avoid the pointer-chase of `fptr->self` if desired. The downside of this approach is that, as you identified, it costs an extra indirection in the implementation of most IO methods to obtain the impl from the VALUE. But, the benefits are that we get almost total encapsulation whilst maintaining backwards compatibility with almost all existing code. We can also shuffle fields between the private & public structs depending on observed compatibility issues found in the wild too. ---------------------------------------- Feature #19057: Hide implementation of `rb_io_t`. https://bugs.ruby-lang.org/issues/19057#change-103322 * Author: ioquatix (Samuel Williams) * Status: Open * Priority: Normal * Assignee: ioquatix (Samuel Williams) ---------------------------------------- In order to make improvements to the IO implementation like <https://bugs.ruby-lang.org/issues/18455>, we need to add new fields to `struct rb_io_t`. By the way, ending types in `_t` is not recommended by POSIX, so I'm also trying to rename the internal implementation to drop `_t` where possible during this conversion. Anyway, we should try to hide the implementation of `struct rb_io`. Ideally, we don't expose any of it, but the problem is backwards compatibility. So, in order to remain backwards compatibility, we should expose some fields of `struct rb_io`, the most commonly used one is `fd` and `mode`, but several others are commonly used. There are many fields which should not be exposed because they are implementation details. ## Current proposal The current proposed change <https://github.com/ruby/ruby/pull/6511> creates two structs: ```c // include/ruby/io.h #ifndef RB_IO_T struct rb_io { int fd; // ... public fields ... }; #else struct rb_io; #endif // internal/io.h #define RB_IO_T struct rb_io { int fd; // ... public fields ... // ... private fields ... }; ``` However, we are not 100% confident this is safe according to the C specification. My experience is not sufficiently wide to say this is safe in practice, but it does look okay to both myself, and @Eregon + @tenderlovemaking have both given some kind of approval. That being said, maybe it's not safe. There are two alternatives: ## Hide all details We can make public `struct rb_io` completely invisible. ```c // include/ruby/io.h #define RB_IO_HIDDEN struct rb_io; int rb_ioptr_descriptor(struct rb_io *ioptr); // accessor for previously visible state. // internal/io.h struct rb_io { // ... all fields ... }; ``` This would only be forwards compatible, and code would need to feature detect like this: ```c #ifdef RB_IO_HIDDEN #define RB_IOPTR_DESCRIPTOR rb_ioptr_descriptor #else #define RB_IOPTR_DESCRIPTOR(ioptr) rb_ioptr_descriptor(ioptr) #endif ``` ## Nested public interface Alternatively, we can nest the public fields into the private struct: ```c // include/ruby/io.h struct rb_io_public { int fd; // ... public fields ... }; // internal/io.h #define RB_IO_T struct rb_io { struct rb_io_public public; // ... private fields ... }; ``` ## Considerations I personally think the "Hide all details" implementation is the best, but it's also the lest compatible. This is also what we are ultimately aiming for, whether we decide to take an intermediate "compatibility step" is up to us. I think "Nested public interface" is messy and introduces more complexity, but it might be slightly better defined than the "Current proposal" which might create undefined behaviour. That being said, all the tests are passing. -- https://bugs.ruby-lang.org/