Issue #22206 has been updated by nobu (Nobuyoshi Nakada). Add dependency updater to `mkmf` to `depend` files conflicts with upstreams. Allow extension projects to update marked dependency sections after `create_makefile` has resolved their source and object lists. Share the dependency scanner with `tool/mkdepend.rb` so bundled and upstream extensions can generate the same dependency files. ---------------------------------------- Misc #22206: Replace `tool/update-deps` with source-based dependency generation https://bugs.ruby-lang.org/issues/22206#change-118167 * Author: nobu (Nobuyoshi Nakada) * Status: Closed ---------------------------------------- I propose replacing `tool/update-deps` with `tool/mkdepend.rb`, a dependency generator written in Ruby that scans the source files directly. The complete dependency lists would be generated into the build directory instead of being committed. The repository would retain only the source mappings and manual rules needed to determine which source builds each object. ## Problems with `tool/update-deps` `tool/update-deps` has several practical problems. 1. It depends on GNU make and GCC-specific output. It obtains build information from `make -p` and requires GCC’s `-save-temps=obj` option. This also means that `ccache` and similar compiler wrappers must be disabled. The dependency update procedure therefore cannot run directly on every platform supported by Ruby. 2. Updating dependencies is expensive. The current procedure requires a configured tree and a compiler build before dependencies can be inspected. A dependency-only change should not require compiling Ruby and its extensions first. 3. The generated `depend` files are very large. Most of their contents are mechanically generated header dependencies. They make reviews noisy, cause large diffs when a commonly included header changes, and are easy to leave stale. In the prototype, approximately 70,000 generated lines can be removed from 98 `depend` files. 4. The checked-in files duplicate information available from the sources. Most header dependencies can be obtained by recursively following `#include` directives. Only the object-to-source mappings and a small number of rules for generated or substituted sources need to be recorded explicitly. ## Proposal Add `tool/mkdepend.rb` and use it instead of `tool/update-deps`. `tool/mkdepend.rb` scans C and parser sources recursively, resolves quoted and angle-bracket includes using Ruby’s source layout, and writes Make dependency rules. It does not require a configured build, GNU make, or a C compiler, and it runs with Ruby 3.1. Unknown preprocessor conditions are handled conservatively by scanning both branches. Source-specific `define` and `undef` declarations are available when one branch must be selected. Small declarations in `depend` files also describe generated source templates, logical dependencies, and macro-based include names that cannot be inferred from ordinary file lookup. Targets declared in the manual part of each `depend` file are treated as generated files. This avoids maintaining a second list of generated files. ### Repository and build directory layout The sections between these markers remain in the repository: ```make # AUTOGENERATED DEPENDENCIES START array.$(OBJEXT): {$(VPATH)}array.c # AUTOGENERATED DEPENDENCIES END ``` They contain only the minimal object-to-source mappings. Manual Make rules outside the section remain unchanged. When a base Ruby is available, `configure` expands these mappings and writes the complete dependency files under `.deps` in the build directory. The generated Makefile includes those files. This applies to out-of-tree builds, GNU make builds, and Windows NMAKE builds. Release snapshots expand the dependencies before packaging. A release tarball can therefore still be configured without a base Ruby and use the dependency files included in the source tree. ### Commands The proposed commands are: ```sh # Check the committed source mappings. ruby tool/mkdepend.rb --scope=all --sources --check # Update the committed source mappings. ruby tool/mkdepend.rb --scope=all --sources --inplace # Generate complete dependencies. ruby tool/mkdepend.rb --scope=all --output=.deps ``` The existing `make check-depends` and `make fix-depends` targets invoke these operations, so contributors do not need to remember the command-line details. ## Benefits - Dependency generation works without GNU make or GCC. - It runs before the main build and does not require compiler output. - Complete dependencies are generated for the current source tree on every build instead of being kept as a large generated snapshot in Git. - Dependency updates become small and reviewable, usually changing only an object-to-source mapping or an explicit declaration. - The same generator can produce paths appropriate for GNU make and NMAKE. - Out-of-tree builds keep generated dependency data out of the source tree. ## Limitations This is a dependency scanner, not a complete C preprocessor. Conditions that cannot be decided safely are scanned in both directions, which may produce extra dependencies but should not omit required ones. The scanner also cannot infer that a newly added C file should become a build object when there is no object rule for it. That information belongs to the Makefile or the minimal source mapping and must still be added explicitly. ## Prototype A working implementation is available at: <https://github.com/nobu/ruby/tree/mkdepend> It includes the source scanner, dependency declarations, configure and NMAKE integration, snapshot handling, dependency checks, and tests for Ruby 3.1. -- https://bugs.ruby-lang.org/