February 2023 - ruby-core - ml.ruby-lang.org

[ruby-core:111712] [Ruby master Feature#19322] Support spawning "private" child processes
by kjtsanaktsidis (KJ Tsanaktsidis) 13 Feb '23

13 Feb '23

Issue #19322 has been reported by kjtsanaktsidis (KJ Tsanaktsidis). ---------------------------------------- Feature #19322: Support spawning "private" child processes https://bugs.ruby-lang.org/issues/19322 * Author: kjtsanaktsidis (KJ Tsanaktsidis) * Status: Open * Priority: Normal ---------------------------------------- ## Background The traditional Unix process APIs (`fork` etc) are poorly isolated. If a library spawns a child process, this is not transparent to the program using the library. Any signal handler for `SIGCHLD` in the program will be called when the spawned process exits, and even worse, if the parent calls `Process.waitpid2(-1)`, it will consume the returned status code, stealing it from the library! Unfortunately, the practice of responding to `SIGCHLD` by calling `waitpid2(-1)` in a loop is a pretty common unixism. For example, Unicorn does it [here](https://yhbt.net/unicorn.git/tree/lib/unicorn/http_server.rb#n401). In short, there is no reliable way for a gem to spawn a child process in a way that can’t (unintentionally) be interfered with by other parts of the program. ## Existing solutions in OS’s Several operating systems provide an improved API for spawning child processes which are fully isolated; that is, they do not generate `SIGCHLD` signals in the program, and are invisible to calls to `waitpid(2)` * On Linux, such invisible processes can be made by calling `clone(2)` with a zero value in the low byte of `flags`. If the CLONE_PIDFD flag is also provided, then a file descriptor representing the process is also returned; this can be used to wait for and signal the process in a race-free way. * On FreeBSD, the `pdfork(2)` syscall makes a process that does not signal SIGCHLD and is ignored by `waitpid(2)` calls that do not explicitly specify the pid (i.e. it is ignored when -1 is passed). It also returns a file descriptor representing the process. Both of these APIs center around the idea of a process file descriptor. Rather than managing a child process using the old process-global wait/signal mechanisms, they return a file descriptor representing the process. Such a file descriptor can uniquely identify the spawned process, be used to wait on the process and get the status, send signals, and even participate in `poll(2)`. They also protect against pid-reuse race conditions; after a process has terminated and been reaped, the pidfd becomes invalid, and can’t randomly begin to refer to a different process. ## Proposed Ruby APIs I think we should make a new API `Process.spawn_handle`, which accepts all the same parameters as `Process.spawn`. However, it does _not_ return a pid like `Process.spawn`, but rather a new type `Process::Handle`. `Process::Handle` would identify a single spawned process, using a durable OS-supplied handle not subject to re-use risks (e.g. a pidfd). It would provide the following methods: * `#pid` - get the pid that the handle is for. * `#send_signal(signal)` - send a signal to the wrapped process (where "signal" is a symbol, string, or number with the same meaning as in `Process.kill`. * `#wait` - blocks waiting for the program to exit, and then returns a `Process::Status` object representing e.g. the exit code. Like calling `waitpid`. * `#wait_nonblock` - Returns a `Process::Status` object for the child process. If the child has not exited, it will be a status object for which `#exited?` is false. Does not block. Like calling `waitpid(WNOHANG)`. Finally, the `Open3` family of methods would be extended to accept `handle:` as an additional keyword argument. When set to true, `Process.spawn_handle` will be used to start the child, and `Process::Handle` objects will be returned in place of pids. Modifying backticks, `Kernel#system` and other process-creating methods which don't return pids to use `spawn_handle` internally would also be possible, but out of scope for an initial implementation of this ticket. ## OS compatibility For this API to be useful to gem authors, it has to be widely available on the systems that they and their users care about. As discussed, the `clone(2)` syscall and `CLONE_PIDFD` flag can be used on Linux 5.2+ to implement `Process::Handle`. FreeBSD has `pdfork(2)` since v9. I haven’t investigated Windows _deeply_, but I think Windows doesn’t really have the notion of process-global `waitpid` or `SIGCHLD` anyway. The `CreateProcess` function returns a `PROCESS_INFORMATION` struct, which returns a `HANDLE` for the child process, which seems analogous to a process FD. However this does leave a large chunk of operating systems which don’t have this functionality built-in. Off the top of my head: * MacOS, NetBSD, and OpenBSD have nothing. I stared pretty hard at the Darwin XNU source and couldn’t find a race-free way to convince it not to dispatch `SIGCHLD` for a particular process or stop it from being reaped by process-wide `wait4` calls. * Linux < 5.2 is in some probably-pretty-widely-deployed-still distros - it’s the release kernel in Ubuntu 18.04 for example. I have two ideas for how the semantics of `Process::Handle` could be emulated on such systems. However I recognise that they aren’t amazing so if anybody has some better ideas I would dearly love to hear them. ### Long-lived proxy The first time `Process.spawn_handle` is used, we would fork/exec a long-lived “fork-helper” program. This could be a separate helper binary we compile with the build system, or perhaps just a re-invocation of the ruby interpreter with something like `ruby -e “Process._fork_helper”`. There would be a unix socketpair shared between the parent process & the helper. Instead of actually forking when we’re calling `Process.spawn_handle`, we would instead send a message on this socket asking the helper to, _itself_, fork & exec the specified program. Any file descriptors etc needed in the child could also be sent over this socket. All of the `Process::Handle` methods would be proxies which called through to the helper binary. This way, the ruby process is never actually the parent of the spawned child, so we would never get any SIGCHLD etc from it. The fork-helper program might generate a SIGCHLD, but it should persist until the ruby process exits; we would only generate a SIGCHLD signal if it crashed abnormally. ### Forward misdirected waits With this approach, `Process.spawn_handle` would just `fork(2)`/`exec(2)` or `posix_spawn(2)` processes like normal. We would however keep a table of pids -> `Process::Handle` instances. When Ruby’s C-level SIGCHLD handler is invoked, we would inspect that table and see if the pid has an associated `Process::Handle`. If so, we would skip calling any registered Ruby SIGCHLD handler; instead, we would call `waitpid` ourselves, update the status info on the handle object, and unblock anybody waiting on `Process::Handle#wait`. Likewise, in the C-side implementation of `Process.waitpid2` etc, we would check the returned pid from the syscall against the handle table. If it matched, we would perform the same work as in the SIGCHLD case, and then re-start the original call to `Process.waitpid2`. This approach keeps the process tree correct and involves less silly proxying, but it won’t hide the process from any callers to the raw `waitpid` library functions in C extensions. Doing that seems like a silly idea anyway though, so maybe that’s OK? ## Motivation My use-case for this is that I’m working on a perf-based profiling tool for Ruby. To get around some Linux capability issues, I want my profiler gem (or CRuby patch, whatever it winds up being!) to fork a privileged helper binary to do some eBPF twiddling. But, if you’re profiling e.g. a Unicorn master process, the result of that binary exiting might be caught by Unicorn itself, rather than my (gem | interpreter feature). In my case, I'm so deep in linux specific stuff that just calling `clone(2)` from my extension is probably fine, but I had enough of a look at this process management stuff I thought it would be worth asking the question if this might be useful to other, more normal, gems. -- https://bugs.ruby-lang.org/

4 13

[ruby-core:111194] [Ruby master Feature#19179] Support parsing SCM_CRED(ENTIALS) messages from ancillary messages
by kjtsanaktsidis (KJ Tsanaktsidis) 13 Feb '23

13 Feb '23

Issue #19179 has been reported by kjtsanaktsidis (KJ Tsanaktsidis). ---------------------------------------- Feature #19179: Support parsing SCM_CRED(ENTIALS) messages from ancillary messages https://bugs.ruby-lang.org/issues/19179 * Author: kjtsanaktsidis (KJ Tsanaktsidis) * Status: Open * Priority: Normal ---------------------------------------- ## Background Linux and FreeBSD support processes at either end of a unix socket identifying themselves to the other party by passing an ancillary message of type `SCM_CREDENTIALS` (Linux) or `SCM_CREDS` (FreeBSD). The socket library contains code to parse these ancillary messages, but the only way this is exposed into Ruby code is by the `Socket::AncillaryData#inspect` method - e.g. ``` # On Linux irb(main):002:0> s1, s2 = UNIXSocket.pair => [#<UNIXSocket:fd 5>, #<UNIXSocket:fd 6>] irb(main):004:0> s2.setsockopt Socket::SOL_SOCKET, Socket::SO_PASSCRED, 1 => 0 # struct ucred on Linux is (32-bit signed) pid_t, followed by (32-bit unsigned) uid_t, followed by # (32-bit unsigned) gid_t irb(main):008:0> ancdata = [Process.pid, Process.uid, Process.gid].pack("lLL") => "\x1ET\x05\x00\xE8\x03\x00\x00\xE8\x03\x00\x00" # Socket::AncillaryData knows how to unmarshal the data into struct ucred irb(main):010:0> ancmsg = Socket::AncillaryData.new(Socket::AF_UNIX, Socket::SOL_SOCKET, Socket::SCM_CRE DENTIALS, ancdata) => #<Socket::AncillaryData: UNIX SOCKET CREDENTIALS pid=349214 uid=1000 gid=1000 (ucred)> irb(main):011:0> s1.sendmsg "hi", 0, nil, ancmsg => 2 # ancillary message can be passed through irb(main):012:0> _, _, _, recvanc = s2.recvmsg; recvanc => #<Socket::AncillaryData: UNIX SOCKET CREDENTIALS pid=349214 uid=1000 gid=1000 (ucred)> ``` On Linux, at least, a suitably privileged process can send any value through for the pid, uid, or gid, but the kernel will reject attempts by unprivileged processes to forge credentials in this way. So SCM_CREDENTIALS messages can be useful for certain systems programming tasks. A somewhat wider array of operating systems support querying the identity of the other side of a socket using a socket option, variously `SO_PEERCRED` (Linux, OpenBSD) or `LOCAL_PEERCRED` (FreeBSD, MacOS). Again, the socket library is able to unmarshal the socket data into the correct structure on these various systems, but it's only exposed to Ruby code via `#inspect` - e.g. ``` irb(main):002:0> s1, s2 = UNIXSocket.pair => [#<UNIXSocket:fd 5>, #<UNIXSocket:fd 6>] irb(main):014:0> s1.getsockopt Socket::SOL_SOCKET, Socket::SO_PEERCRED => #<Socket::Option: UNIX SOCKET PEERCRED pid=349214 euid=1000 egid=1000 (ucred)> ``` Ruby _does_ however support e.g. `BasicSocket#getpeereid`, which could use `SO_PEERCRED` etc under the hood - so getting the uid/gid data is not totally impossible. I believe getting the pid is though. ``` irb(main):016:0> s1.getpeereid => [1000, 1000] ``` ## My proposal I believe we should implement the following: * `Socket::Credentials` - this would be a struct which can contain all the various platform-specific pieces of credential info that can be transferred over a socket, such as uid, gid, pid, euid, egid, and group list. * `Socket::AncillaryData#credentials` - this would parse an `SCM_CREDS` or `SCM_CREDENTIALS` ancillary data message into the appropriate platform-specific struct, and return a `Socket::Credentials` instance containing that data. This would be analogous to `Socket::AncillaryData#int`; a method for interpreting the ancillary data in a certain form. * `Socket::Option#credentials` - This would parse a `SO_PEERCRED` or `LOCAL_PEERCRED` socket option response into the appropriate platform-specific struct, and return a `Socket::Credentials` instance containing that data. Again, this would be analogous to `Socket::Option#int`. The existing `struct ucred`/`struct xucred`/`struct sockpeercred`/`struct cmsgcred` parsing code (used only for `#inspect` output) would be moved into `Socket::Credentials`, and `Socket::AncillaryData#inspect`/`Socket::Option#inspect` would be implemented in terms of `Socket::Credentials`. This would nicely wrap a lot of parsing work that Ruby is already doing, into an API which allows Ruby code to take advantage of it. ## Use-cases My motivation for designing this feature came about whilst I was experimenting with some ideas for Ruby profilers. I wanted to allow a CLI tool to ask a Ruby process to start profiling itself by sending a message on a unix socket. Alongside the message, it would send a file descriptor which was the result of calling `perf_event_open(2)` in the CLI tool. In order to call `perf_event_open(2)`, the CLI tool would need to be privileged. I also wanted the Ruby process to authenticate the request and make sure it came from the same UID that it was running as. Calling `BasicSocket#getpeereuid` would reveal the remote process to be running as UID 0, (or perhaps even some other UID, with sufficient ambient capabilities to call `perf_event_open`). Instead, I decided to make the CLI tool send a `SCM_CREDENTIALS` message containing the uid of the process to be profiled; that way, the kernel does all the policy checking on whether or not this is actually allowed, and the Ruby process receiving the message just needs to check if `uid == Process.getuid`. I think, on Linux at least, that this feature will be useful for any kind of communication/authentication scheme between privileged & unprivileged processes over unix sockets. ## My implementation I have an implementation of roughly this in this pull request: https://github.com/ruby/ruby/pull/6822 Thanks! -- https://bugs.ruby-lang.org/

3 4

[ruby-core:111614] [Ruby master Feature#19306] Expand zlib interface
by ioquatix (Samuel Williams) 13 Feb '23

13 Feb '23

Issue #19306 has been reported by ioquatix (Samuel Williams). ---------------------------------------- Feature #19306: Expand zlib interface https://bugs.ruby-lang.org/issues/19306 * Author: ioquatix (Samuel Williams) * Status: Open * Priority: Normal ---------------------------------------- Introduce some extra methods like `window_size`, `level` (compression/decompression), enhance `inspect` and various other quality of life improvements. -- https://bugs.ruby-lang.org/

2 2

[ruby-core:112381] [Ruby master Misc#16895] Request for cooperation: Try your applications/libraries with master branch and debug options
by rubyFeedback (robert heiler) 12 Feb '23

12 Feb '23

Issue #16895 has been updated by rubyFeedback (robert heiler). As I got older (it has been almost 20 years now since I have been using ruby, and I feel physically older too compared to the more young-version of me back in the days), my focus has shifted a bit. I was super-experimental when I was younger. Nowadays I try to stay with stable ruby releases whenever possible. So the xmas-release is usually when I switch; I do tend to switch as quickly as possible, to avoid any legacy-drag that we had from e. g. ruby 1.8.x to 1.9.x to 2.x and beyond. I am kind of like "an early adopter of stable ruby", if that makes sense. :D I can still find some issues here and there and report it. Testing unstable is a bit of an issue and hassle, though. I wrote a lot of ruby scripts over the years, and ruby is used to maintain my whole linux system (I compile from source whenever possible); so getting the new git release of ruby is as simple as doing "ry ruby --gitty" for me. That will download the latest git sources, re-package them into .tar.xz and put it into $SRC_DIR/ruby/. I can even automatically compile it too. So this is actually the simple part. The bigger issue is time investment, and forgetting it. I kind of adjusted mentally to be lazy, so I download the latest ruby on xmas and then keep on using it, until the next xmas. So I rarely get to test new features, unless they are interesting. I don't have a good alternative model. Would be interesting if we could have ruby so modular that there be tons of ruby dialects, and we could test each dialect in the "middle of the year", just recompiling parts of ruby that would be affected (e. g. like when pattern matching was added. Imagine if we could have a toggle for all of ruby behaviour, a bit like we can specify the linux kernel). ---------------------------------------- Misc #16895: Request for cooperation: Try your applications/libraries with master branch and debug options https://bugs.ruby-lang.org/issues/16895#change-101823 * Author: ko1 (Koichi Sasada) * Status: Open * Priority: Normal ---------------------------------------- ## Summary If you maintain a Ruby application or library, please consider testing with the Ruby `master` branch, including the *debug* build. This should be in addition to testing with all supported stable releases of Ruby. To make this easier, we are providing Docker images and GitHub actions, as outlined below. ## Details The rapid pace of Ruby development sometimes introduces bugs, such as incorrect behaviour or unexpected incompatibilities. Despite our best efforts and testing, without your feedback, we cannot catch every issue. Understanding how our changes impact downstream code is important feedback for the Ruby core developers. We want to know how your applications work on master. If you encounter an error when testing with the `master` branch (e.g. `[BUG] ...` in output log), [please report it](https://bugs.ruby-lang.org). It will be very helpful. ### Testing With `master` Testing using the `master` branch (sometimes referred to as `ruby-head`) will make your Ruby scripts ready for the next Ruby version. It also helps us catch incompatibilities as we change and evolve Ruby's public interface. ### Testing With Debug Build Testing with the master branch debug build enables many assertions within the Ruby interpreter. These assertions can detect incorrect usage of the C extensions, and also bugs in the interpreter when running your program. These assertions have an impact on the performance of the interpreter. To compile a debug build, refer the later section titled "Building With Debug Mode". ## Continuous Integration With `master` Building Ruby for your own testing environment can be difficult, so we are providing two convenient ways to use the master branch in your existing testing pipeline: - Docker Images - Github Action ### Docker Images The [rubylang docker repository](https://hub.docker.com/r/rubylang/ruby) provides images for various Ruby versions, including nightly builds of master with and without debug assertions" - Nightly built master: `rubylang/ruby:master-nightly-bionic` - Nightly debug built master: `rubylang/ruby:master-debug-nightly-bionic` Here is an example `Dockerfile`: ```dockerfile FROM rubylang/ruby:master-nightly-bionic ``` Then to build: ```shell $ docker build . Sending build context to Docker daemon 2.048kB Step 1/1 : FROM rubylang/ruby:master-nightly-bionic master-nightly-bionic: Pulling from rubylang/ruby ... Status: Downloaded newer image for rubylang/ruby:master-nightly-bionic ---> 059d367a8fbd Successfully built 059d367a8fbd ``` ### GitHub Action The [GitHub Action to setup Ruby](https://github.com/ruby/setup-ruby) provides both `ruby-head` and `ruby-debug` builds. Here is an example workflow to test on all Ruby stable releases, including `ruby-head` and `ruby-debug`: ```yaml name: Development on: [push] jobs: test: strategy: fail-fast: false matrix: os: [ubuntu] ruby: [2.5, 2.6, 2.7, head, debug] runs-on: ${{ matrix.os }}-latest continue-on-error: ${{ matrix.ruby == 'head' || matrix.ruby == 'debug' }} steps: - uses: actions/checkout@v2 - uses: ruby/setup-ruby@v1 with: ruby-version: ${{ matrix.ruby }} - run: bundle install - run: bundle exec rake ``` See the [documentation](https://github.com/marketplace/actions/setup-ruby-jruby-and-… for more details on how to use this action. ## Building With Debug Mode To create a debug build of Ruby, execute the following commands: ``` $ git clone https://github.com/ruby/ruby.git $ cd ruby $ autoconf $ cppflags=-DRUBY_DEBUG=1 ./configure --prefix=$HOME/.rubies/ruby-debug $ make install ``` If you are using `chruby`, you can switch to the above build: ``` $ chruby ruby-debug ``` You can find more details to build Ruby master in the [README](https://github.com/ruby/ruby/blob/master/README.md#how-to-compile-a…. ## Acknowledgement We thank @mrkn for the Docker image, @eregon for the GitHub Action, and @ioquatix for reviewing this announcement. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:112380] [Ruby master Misc#16507] =~ vs include? or match?
by rubyFeedback (robert heiler) 12 Feb '23

12 Feb '23

Issue #16507 has been updated by rubyFeedback (robert heiler). I can't speak for the ruby core team, but here is my rule-of-thumb in my own ruby code "bases" out there. I usually try to stick with the simplest choice possible, which in many ways is .include?() for me. Sometimes this is not possible, though, such as when the input is not so simple (e. g. where a regex is more appropriate such as /\d{0,6}\.\d{3,6}/ like when the input can vary). Regexes are great in ruby; I think matz liked them from the perl days. :) I almost never use .match() or .match?() - and I think these are also rare in other people's code bases out there. I tend to use methods ending via "?" a lot. It is a bit like DSL-design for me: if object.can_do_this? or object.can_do_that? end I think it reads very nice. That is actually my recommendation too - if you feel the readability is improved then use the methods. The method .ci?() seems a bit short-ish. I would recommend against such short method names; they rarely seem to improve a lot. I tend to use them mostly when I need to do things where being succinct is important. .mswin?() makes more sense. Even then I would perhaps consider using two words, if that makes sense for the code you are working with. In my opinion, when it really helps you maintain the code, and when it leads to some clarity, using helper-methods is a good idea. Perhaps if you want some more feedback from ruby core devs, it may help if you could give a few more specific examples so they can look at the instances where you thought that removing =~ would be a good idea. Sometimes different ruby developers have a different focus and priority for using certain constructs in their code. For me my own code is very simple and readable, but I am sure for others it can be very messy and unreadable (I try to stick to simplicity at all times and avoid too complicated constructs; for instance I never use "->" and other things, but I use blocks a LOT, so my code will have tons of "if block_given?; yield = yielded" just to keep on working with the variable. Blocks are great). ---------------------------------------- Misc #16507: =~ vs include? or match? https://bugs.ruby-lang.org/issues/16507#change-101822 * Author: MSP-Greg (Greg L) * Status: Open * Priority: Normal ---------------------------------------- While working on getting the mswin build working on Actions, I thought adding mswin? and ci? methods to Minitest::Unit::Guard in tool/lib/minitest/unit.rb would be helpful. Currently some tests are failing that are guarded/skipped based on ENV['APPVEYOR']. For ci?, I'd combine Travis, AppVeyor, & Actions. There are quite a few instances where =~ is used for a boolean return. Would it be considered appropriate to replace those calls with include? or match? -- https://bugs.ruby-lang.org/

1 0

[ruby-core:112377] [Ruby master Feature#14982] Improve namespace system in ruby to avoiding top-level names chaos
by shioyama (Chris Salzberg) 12 Feb '23

12 Feb '23

Issue #14982 has been updated by shioyama (Chris Salzberg). For those following this issue, it is now possible in Ruby 3.2. to write autoloaded code that does not touch the global namespace, with Im, a fork of Zeitwerk I have been working on over the past couple months: https://github.com/shioyama/im Im relies on two new features in 3.2: `Kernel#load` with a module as second argument, and `Module#const_added`. Further details are provided in the readme, but the tl;dr is that it provides an interface almost identical to Zeitwerk, but rather than autoloading to `Object` it autoloads to an anonymous module (an instance of `Im::Loader`, which is a subclass of `Module`.) I have also developed a demo Rails app which uses Im to load the entire application onto a single namespace, while allowing you to actually write the code as you normally would, at toplevel. https://github.com/shioyama/rails_on_im ---------------------------------------- Feature #14982: Improve namespace system in ruby to avoiding top-level names chaos https://bugs.ruby-lang.org/issues/14982#change-101819 * Author: jjyr (Jinyang Jiang) * Status: Open * Priority: Normal * Assignee: matz (Yukihiro Matsumoto) ---------------------------------------- Updated: https://bugs.ruby-lang.org/issues/14982#note-5 ## Why Ruby has evaluation all class/module names in top-level context(aka TOPLEVEL_BINDING). As a user we basically hard to know how many names in the current context, is causing chaos in some cases. For example: case 1: Put common used errors class in a single file, like below ``` ruby # utils/errors.rb class FooError end class BarError end ``` In other files under 'utils' we want to use those errors, so the best practice is to use `require_relative 'errors'` in each file we need. ``` ruby # utils/binary_helper.rb # we forget require errors module BinaryHelper # ... raise BarError # ... end ``` But sometime we may forget to require dependencies in a file, it's hard to notice because if RubyVM already execute the requires we still can access the name BarError, but if user directly to require 'utils/binary_helper', he/she will got an NameError. case 2: Two gems use same top-level module name, so we can't use them together ## The Reason of The Problem The reason is we let module author to decision which module user can use. ('require' is basically evaluation, highly dependent on the module author's design) But we should let users control which names to use and available in context. As many other popular languages dose(Rust, Python..) I think the solution is basically the same philosophy compares to refinement feature. ## The Design I propose an improved namespace to Ruby, to solve the problems and still compatible with the current Ruby module system. ``` ruby class Foo end # introduce Kernel#namespace namespace :Hello do # avoiding namespace chaos # Foo -> NameError, can't access TOPLEVEL_BINDING directly # Kernel#import method, introduce Foo name from TOPLEVEL_BINDING import :Foo # in a namespace user can only access imported name Foo # import constant to another alias name # can avoid writing nested module/class names import :"A::B::C::D", as: :E # require then import, for convenient import :"A::B::C::D", as: :E, from: 'some_rb_file' # import same name from two gems import :"Foo", as: :Foo_A, from: 'foo_a' import :"Foo", as: :Foo_B, from: 'foo_b' # import names in batch import %i{"A::B::C::D", "AnotherClass"}, from: 'some_rb_file' # import and alias in batch import {:"A::B::C::D" => :E, :Foo => Foo2}, from: 'some_rb_file' class Bar def xxx # can access all names in namespace scope [Foo, Foo_A, Foo_B] end end end Hello.class # -> module. namespace is just a module Hello::Bar # so we do not broken current ruby module design # namespace system is intent to let user to control names in context # So user can choose use the old require way require 'hello' Hello::Bar # Or user can use namespace system as we do in hello.rb namespace :Example do import :"Hello::Bar", as: :Bar Bar # ok Foo # name error, cause we do not import Foo in :Example namespace end Foo # ok, cause Foo is loaded in TOPLEVEL_BINDING # define nested namespace # more clear syntax than “module Example::NestedExample” namespace :NestedExample, under: Example do end namespace :Example2 do namespace :NestedExample do end end ``` Pros: * Completely compatible with the current module system, a gem user can completely ignore whether a gem is write in Namespace or not. * User can completely control which names in current context/scope. * May solve the top module name conflict issue(depends on VM implementation). * Avoid introducing new keyword and syntax. * Type hint or name hint can be more accuracy under namespace(not sure). Cons: * Need to modify Ruby VM to support the feature. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:112239] [Ruby master Feature#19420] Simplify MJIT implementation
by k0kubun (Takashi Kokubun) 11 Feb '23

11 Feb '23

Issue #19420 has been reported by k0kubun (Takashi Kokubun). ---------------------------------------- Feature #19420: Simplify MJIT implementation https://bugs.ruby-lang.org/issues/19420 * Author: k0kubun (Takashi Kokubun) * Status: Open * Priority: Normal ---------------------------------------- # Background ### Implementation complexity Because MJIT needs to run a C compiler and be able to call almost everything in vm.c, it has brought MJIT-specific complexity to the Ruby VM implementation: * GVL and waitpid hacks for `Process.waitall` * @ko1 and @ioquatix have had problems dealing with it. * `vm_base_ptr` dependency from MJIT * @ko1 wanted to remove `cfp->__bp__` commit:8dd0fb9039bbe6152ea5542e6bc70de152871e23 * This is technically not the fault of the MJIT design, but the current implementation relies on it. * MJIT header and `MJIT_HEADER` macro * It seems like some build dependency is missing for it and my local build with `-j` randomly fails on MJIT header build. * MJIT-specific frame handling * `catch_except_p` flag in ISeq * `jit_enable_p` flag in `vm_exec` * `MJIT_FUNC_EXPORTED` functions ### Third-party JIT experiments Despite YJIT being the only non-experimental JIT compiler in CRuby from Ruby 3.2, people still experiment with other JIT implementations, e.g. [vnmakarov/ruby (sir-mirjit)](https://github.com/vnmakarov/ruby) [tenderlove/tenderjit](https://github.com/tenderlove/tenderjit), [jhawthorn/hawthjit](https://github.com/jhawthorn/hawthjit), which I think is a good thing for the future of Ruby, and for YJIT as well. In Ruby 3.2, MJIT was rewritten in Ruby. It also meant that you could replace CRuby's JIT implementation by monkey-patching MJIT, which became the first ever "third-party JIT backdoor". I know two people started using it for that purpose. I want people (including myself through MJIT) to keep experimenting with Ruby performance, so I don't want to just get rid of this. `RubyVM::MJIT::C` and `RubyVM::MJIT::Hooks` are something that third-party gems can't easily provide. They should be considered as something similar to JVMCI (Java's JIT interface) whether we make them private or not. # Proposal * Change MJIT to a simple JIT implementation that doesn't rely on a C compiler at runtime, which works like YJIT (to remove all the complexity mentioned above) but is written in Ruby (to keep offerring modules for third-party JITs). * I have [a PoC branch](https://github.com/k0kubun/ruby/compare/07c19cf551e58e1b82af5efeb00…, which passes `make check`. It currently supports very limited operations and is not really useful, but I intend to keep improving this outside my work hours (that I dedicate to YJIT). Note that we could still solve the above problems even with the current patch. * It shouldn't increase the maintenance cost of CRuby too much. You're free to just mark something "can't compile" in MJIT when VM is changed. It should also not support platforms that YJIT doesn't support. * It's also kind of useful in the sense that it requires neither a C compiler at runtime nor a Rust compiler at build time. -- https://bugs.ruby-lang.org/

3 2

[ruby-core:112359] [Ruby master Feature#10343] Postfix notations for `when` and `else` inside `case` statement
by rubyFeedback (mark potter) 11 Feb '23

11 Feb '23

Issue #10343 has been updated by rubyFeedback (mark potter). I do not necessarily have any opinion in favour, but neither in disfavour. Kazuki's old comment probably refers primarily to syntax-wise how the "when" do not align, so on that note aligned "when" would look prettier. Actually the trailing "else" is the most confusing part to me; the "when" does not seem that awkward, even though it is a bit. It may be interesting to ask matz's opinion nonetheless either way - if he were to start ruby from scratch, would he opt for such a syntax; after all we have had other additions such as endless method definitions or pattern matching. :) (I for one would love to see case/when become a more "first-class citizen" feature, where we could query its internal state at all times, or use different case/when to be combined, a bit like treating them as objects with specific state and behaviour - but I guess I should add a new request rather than repurpose Tsuyoshi's issue here.) ---------------------------------------- Feature #10343: Postfix notations for `when` and `else` inside `case` statement https://bugs.ruby-lang.org/issues/10343#change-101802 * Author: sawa (Tsuyoshi Sawada) * Status: Open * Priority: Normal ---------------------------------------- In a `case` statement, the condition part (`when ...`) in the branches do not have the same length in general, and especially, `else` is much shorter than the conditions. So when we write the condition and the return value in a single line, they are not aligned, and are hard to read. ```ruby case foo when some_very_long_proc then "a" when short_regex then "bbb" when some_long_regex then "cc" else "dddd" end ``` I propose to allow postfix notations with `when` and `else` (or `otherwise`) inside `case` statement as below: ```ruby case foo "a" when some_very_long_proc "bbb" when short_regex "cc" when some_long_regex "dddd" else end ``` Pros are: 1) Postfix notation does not require `then` or `;`, so it is concise. 2) The return values from the branches (e.g., `"a"`, `"bb"`, `"ccc"`, `"dddd"`) tend to be shorter and more uniformly lengthened than the conditions, hence they are somewhat close to being aligned naturally, making this easier to read. 3) We are usually more interested in the return value than the condition of a branch, especially when we are reading someone's code and are trying to grasp what the `case` statement does or returns. 4) This notation is closer to case-like conditional notations regularly used in mathematics: ``` ┌ 1 (x = 0) x! = ┤ └ x (x - 1)! (otherwise) ``` So it would be easier to read for those who are familiar with mathematics. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:112355] [Ruby master Bug#19144] Ruby should set AI_V4MAPPED | AI_ADDRCONFIG getaddrinfo flags by default
by kjtsanaktsidis (KJ Tsanaktsidis) 11 Feb '23

11 Feb '23

Issue #19144 has been updated by kjtsanaktsidis (KJ Tsanaktsidis). OK, I opened https://github.com/ruby/ruby/pull/7295 with those changes. Thanks again! ---------------------------------------- Bug #19144: Ruby should set AI_V4MAPPED | AI_ADDRCONFIG getaddrinfo flags by default https://bugs.ruby-lang.org/issues/19144#change-101798 * Author: kjtsanaktsidis (KJ Tsanaktsidis) * Status: Feedback * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN ---------------------------------------- Currently, DNS lookups made with `getaddrinfo` from Ruby (i.e. not from the `Resolv` module) cause both A and AAAA DNS requests to be made, even on systems that don’t actually have an IPv6 address that could possibly make the AAAA response useful. I wouldn’t _really_ care about this, normally, but glibc has a bug (https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1961697) which can cause a 5-second delay in DNS lookups when both A and AAAA records are queried in parallel. This bug is fixed in glibc upstream but still present in some LTS linux distros (Ubuntu 18.04 and 20.04 at least), so I think it’s worthwhile to try and work around it in circumstances where the AAAA request is pointless anyway. The dual A/AAAA lookup happens because whenever Ruby calls getaddrinfo to perform DNS lookups, it always sets `hints`, and sets `hints->ai_flags` to zero by default unless flags are specified by the caller (e.g. `AI_PASSIVE` is set when binding a TCP server socket in `TCPServer.new`). This matches the default value of `ai_flags` specified by POSIX, which is zero. However, glibc behaves differently. When glibc’s `getaddrinfo` function is called with `NULL` for the `hints` parameter, it defaults the `ai_flags` value to `(AI_V4MAPPED | AI_ADDRCONFIG)`. The manpage (from the Linux man-pages project - https://man7.org/linux/man-pages/man3/getaddrinfo.3.html) claims “this is an improvement on the standard” (although I couldn’t find this mentioned in the glibc manual itself). Of course, we’re not _actually_ ever calling `getaddrinfo` with NULL `hints`; so, we never actually use these flags on glibc systems (unless they’re explicitly specified by the caller). My proposal is that we should change Ruby to set these two flags by default, when they’re available, in the following circumstances: * In all calls made internally to `rsock_getaddrinfo` as a result of socket functions like `TCPSocket.new`, `UDPSocket.new`, etc. * EXCEPT when `AI_PASSIVE` is also set (i.e. when we’re trying to get an address to bind for listener socket - see below) * In calls made to `rsock_getaddrinfo` as a direct result of calling `Addrinfo.getaddrinfo` from Ruby with nil flags * EXCEPT calls to `Addrinfo.getaddrinfo` where explicit flags are provided Both of these seem like something you would almost always want to be doing in any outgoing connection scenario: * `AI_V4MAPPED` ensures that, if AF_INET6 is explicitly specified as the desired protocol, and there is no AAAA record in DNS, that any A record that _is_ present gets converted to an IPv4-mapped IPv6 address so it can be used e.g. with NAT64. * `AI_ADDRCONFIG` ensures that, if a machine has no IPv6 address, it doesn’t bother making an AAAA lookup that will return IPv6 addresses that can’t actually be used for anything (and vice versa for IPv4). The reason why we wouldn’t want to set `AI_ADDRCONFIG` in circumstances where Ruby currently sets `AI_PASSIVE` is that loopback addresses are not considered in deciding if a system has an IPv4/IPv6 address. Conceivably, you might want to bind to a `::1` loopback address, and allow other processes on the same machine to connect to that. Does changing this default sound reasonable? If so I can prepare a patch. Another option I considered is doing this _only_ when Ruby is built against glibc (so that other system behaviour is most closely matched). -- https://bugs.ruby-lang.org/

1 0

[ruby-core:112051] [Ruby master Bug#19380] IO.read doesn't validate the offset argument
by andrykonchin (Andrew Konchin) 10 Feb '23

10 Feb '23

Issue #19380 has been reported by andrykonchin (Andrew Konchin). ---------------------------------------- Bug #19380: IO.read doesn't validate the offset argument https://bugs.ruby-lang.org/issues/19380 * Author: andrykonchin (Andrew Konchin) * Status: Open * Priority: Normal * ruby -v: 3.1.3 * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- I've noticed that when passed negative `offset` argument `IO.read` raises a bit confusing error: ```ruby IO.read("read-test.txt", 1, -1) # (irb):55:in `read': Invalid argument @ rb_io_seek - read-test.txt (Errno::EINVAL) ``` But error for invalid `length` argument is much more helpful: ```ruby IO.read("read-test.txt", -1) # (irb):56:in `read': negative length -1 given (ArgumentError) ``` I would expect a similar ArgumentError exception to be raised for invalid `offset` argument instead of `Invalid argument @ rb_io_seek - read-test.txt (Errno::EINVAL)` -- https://bugs.ruby-lang.org/

2 1