[ruby-core:115388] [Ruby master Feature#20005] Add C API to return symbols of native extensions resolved from features
Issue #20005 has been reported by tagomoris (Satoshi Tagomori). ---------------------------------------- Feature #20005: Add C API to return symbols of native extensions resolved from features https://bugs.ruby-lang.org/issues/20005 * Author: tagomoris (Satoshi Tagomori) * Status: Open * Priority: Normal ---------------------------------------- I want an API to resolve symbols of other native extensions by a feature name and a symbol name (just like `dlsym`). (`rb_dln_resolve_symbol` is an example function name of this feature in the example below) ```c // "a_client_open" is a function defined in "/..../a_client.so", that should be loaded in Ruby beforehand. // in the extension to be "a.so" VALUE (*a_client_open)(VALUE); void Init_a(void) { a_client_open = (VALUE(*)(VALUE))rb_dln_resolve_symbol("a_client", "a_client_open"); // the return value may be NULL if the symbols is not found } ``` This API is to replace direct reference of external symbols. Currently, native extensions have to just call the function of other extensions directly on the assumption of: * The dependency native extension is already loaded * The symbol is defined correctly in the dependency (the version of dependency is correct&expected) Otherwise, it crashes. This API provides a way for developers to raise exceptions if the resolved symbol is `NULL`, and these exceptions can display meaningful messages and instructions to the users of extensions. -- https://bugs.ruby-lang.org/
Issue #20005 has been updated by nobu (Nobuyoshi Nakada). `a_client_open` is a function just defined in a_client.so? I.e., do you intend that all non-static functions can be retrieved from other extension libraries without any action in the defining library? ---------------------------------------- Feature #20005: Add C API to return symbols of native extensions resolved from features https://bugs.ruby-lang.org/issues/20005#change-105329 * Author: tagomoris (Satoshi Tagomori) * Status: Open * Priority: Normal ---------------------------------------- I want an API to resolve symbols of other native extensions by a feature name and a symbol name (just like `dlsym`). (`rb_dln_resolve_symbol` is an example function name of this feature in the example below) ```c // "a_client_open" is a function defined in "/..../a_client.so", that should be loaded in Ruby beforehand. // in the extension to be "a.so" VALUE (*a_client_open)(VALUE); void Init_a(void) { a_client_open = (VALUE(*)(VALUE))rb_dln_resolve_symbol("a_client", "a_client_open"); // the return value may be NULL if the symbols is not found } ``` This API is to replace direct reference of external symbols. Currently, native extensions have to just call the function of other extensions directly on the assumption of: * The dependency native extension is already loaded * The symbol is defined correctly in the dependency (the version of dependency is correct&expected) Otherwise, it crashes. This API provides a way for developers to raise exceptions if the resolved symbol is `NULL`, and these exceptions can display meaningful messages and instructions to the users of extensions. -- https://bugs.ruby-lang.org/
Issue #20005 has been updated by tagomoris (Satoshi Tagomori). @nobu No, I intended that the API retrieves `extern`-ed (or `EXPORTS`-ed in *.def) symbols. I missed to mention it. ---------------------------------------- Feature #20005: Add C API to return symbols of native extensions resolved from features https://bugs.ruby-lang.org/issues/20005#change-105330 * Author: tagomoris (Satoshi Tagomori) * Status: Open * Priority: Normal ---------------------------------------- I want an API to resolve symbols of other native extensions by a feature name and a symbol name (just like `dlsym`). (`rb_dln_resolve_symbol` is an example function name of this feature in the example below) ```c // "a_client_open" is a function defined in "/..../a_client.so", that should be loaded in Ruby beforehand. // in the extension to be "a.so" VALUE (*a_client_open)(VALUE); void Init_a(void) { a_client_open = (VALUE(*)(VALUE))rb_dln_resolve_symbol("a_client", "a_client_open"); // the return value may be NULL if the symbols is not found } ``` This API is to replace direct reference of external symbols. Currently, native extensions have to just call the function of other extensions directly on the assumption of: * The dependency native extension is already loaded * The symbol is defined correctly in the dependency (the version of dependency is correct&expected) Otherwise, it crashes. This API provides a way for developers to raise exceptions if the resolved symbol is `NULL`, and these exceptions can display meaningful messages and instructions to the users of extensions. -- https://bugs.ruby-lang.org/
Issue #20005 has been updated by kjtsanaktsidis (KJ Tsanaktsidis). Maybe I’m just not very clever, but I don’t understand why `dlsym` doesn’t already do this? Why doesn’t it work for your use case? I don’t think I understand. ---------------------------------------- Feature #20005: Add C API to return symbols of native extensions resolved from features https://bugs.ruby-lang.org/issues/20005#change-105331 * Author: tagomoris (Satoshi Tagomori) * Status: Open * Priority: Normal ---------------------------------------- I want an API to resolve symbols of other native extensions by a feature name and a symbol name (just like `dlsym`). (`rb_dln_resolve_symbol` is an example function name of this feature in the example below) ```c // "a_client_open" is a function defined in "/..../a_client.so", that should be loaded in Ruby beforehand. // in the extension to be "a.so" VALUE (*a_client_open)(VALUE); void Init_a(void) { a_client_open = (VALUE(*)(VALUE))rb_dln_resolve_symbol("a_client", "a_client_open"); // the return value may be NULL if the symbols is not found } ``` This API is to replace direct reference of external symbols. Currently, native extensions have to just call the function of other extensions directly on the assumption of: * The dependency native extension is already loaded * The symbol is defined correctly in the dependency (the version of dependency is correct&expected) Otherwise, it crashes. This API provides a way for developers to raise exceptions if the resolved symbol is `NULL`, and these exceptions can display meaningful messages and instructions to the users of extensions. -- https://bugs.ruby-lang.org/
Issue #20005 has been updated by nobu (Nobuyoshi Nakada). @kjtsanaktsidis `dlsym` can do, but - not all platforms use `dlopen` and `dlsym`. - `dlsym` needs the handle of the loaded library, and it is not trivial to retrieve the path from the feature name. ---------------------------------------- Feature #20005: Add C API to return symbols of native extensions resolved from features https://bugs.ruby-lang.org/issues/20005#change-105332 * Author: tagomoris (Satoshi Tagomori) * Status: Open * Priority: Normal ---------------------------------------- I want an API to resolve symbols of other native extensions by a feature name and a symbol name (just like `dlsym`). (`rb_dln_resolve_symbol` is an example function name of this feature in the example below) ```c // "a_client_open" is a function defined in "/..../a_client.so", that should be loaded in Ruby beforehand. // in the extension to be "a.so" VALUE (*a_client_open)(VALUE); void Init_a(void) { a_client_open = (VALUE(*)(VALUE))rb_dln_resolve_symbol("a_client", "a_client_open"); // the return value may be NULL if the symbols is not found } ``` This API is to replace direct reference of external symbols. Currently, native extensions have to just call the function of other extensions directly on the assumption of: * The dependency native extension is already loaded * The symbol is defined correctly in the dependency (the version of dependency is correct&expected) Otherwise, it crashes. This API provides a way for developers to raise exceptions if the resolved symbol is `NULL`, and these exceptions can display meaningful messages and instructions to the users of extensions. -- https://bugs.ruby-lang.org/
Issue #20005 has been updated by kjtsanaktsidis (KJ Tsanaktsidis). True, dlopen/dlsym have some portability issues, but:
dlsym needs the handle of the loaded library, and it is not trivial to retrieve the path from the feature name.
Can’t you just call `dlopen(NULL)` to get this? From the Linux man page:
If filename is NULL, then the returned handle is for the main program. When given to dlsym(3), this handle causes a search for a symbol in the main program, followed by all shared objects loaded at program startup, and then all shared objects loaded by dlopen() with the flag RTLD_GLOBAL.
Ruby uses RTLD_GLOBAL to load extensions I think, so any symbols exported by extensions should be accessible ---------------------------------------- Feature #20005: Add C API to return symbols of native extensions resolved from features https://bugs.ruby-lang.org/issues/20005#change-105333 * Author: tagomoris (Satoshi Tagomori) * Status: Open * Priority: Normal ---------------------------------------- I want an API to resolve symbols of other native extensions by a feature name and a symbol name (just like `dlsym`). (`rb_dln_resolve_symbol` is an example function name of this feature in the example below) ```c // "a_client_open" is a function defined in "/..../a_client.so", that should be loaded in Ruby beforehand. // in the extension to be "a.so" VALUE (*a_client_open)(VALUE); void Init_a(void) { a_client_open = (VALUE(*)(VALUE))rb_dln_resolve_symbol("a_client", "a_client_open"); // the return value may be NULL if the symbols is not found } ``` This API is to replace direct reference of external symbols. Currently, native extensions have to just call the function of other extensions directly on the assumption of: * The dependency native extension is already loaded * The symbol is defined correctly in the dependency (the version of dependency is correct&expected) Otherwise, it crashes. This API provides a way for developers to raise exceptions if the resolved symbol is `NULL`, and these exceptions can display meaningful messages and instructions to the users of extensions. -- https://bugs.ruby-lang.org/
Issue #20005 has been updated by nobu (Nobuyoshi Nakada). kjtsanaktsidis (KJ Tsanaktsidis) wrote in #note-5:
True, dlopen/dlsym have some portability issues, but:
dlsym needs the handle of the loaded library, and it is not trivial to retrieve the path from the feature name.
Can’t you just call `dlopen(NULL)` to get this? From the Linux man page:
It returns `RTLD_DEFAULT`, no `dlopen` call is needed. This special handle searches all loaded libraries in the loaded order. That means it can't find a shadowed symbol. Additionally, macOS man page says "This can be a costly search and should be avoided." And another portability concern is exported symbols may be prefixed with "_". Anyway it doesn't look a nice idea to make extensions libraries write `#ifdef _WIN32` unnecessarily here and there. I consider the wrapper API would be reasonable for these reasons. ---------------------------------------- Feature #20005: Add C API to return symbols of native extensions resolved from features https://bugs.ruby-lang.org/issues/20005#change-105334 * Author: tagomoris (Satoshi Tagomori) * Status: Open * Priority: Normal ---------------------------------------- I want an API to resolve symbols of other native extensions by a feature name and a symbol name (just like `dlsym`). (`rb_dln_resolve_symbol` is an example function name of this feature in the example below) ```c // "a_client_open" is a function defined in "/..../a_client.so", that should be loaded in Ruby beforehand. // in the extension to be "a.so" VALUE (*a_client_open)(VALUE); void Init_a(void) { a_client_open = (VALUE(*)(VALUE))rb_dln_resolve_symbol("a_client", "a_client_open"); // the return value may be NULL if the symbols is not found } ``` This API is to replace direct reference of external symbols. Currently, native extensions have to just call the function of other extensions directly on the assumption of: * The dependency native extension is already loaded * The symbol is defined correctly in the dependency (the version of dependency is correct&expected) Otherwise, it crashes. This API provides a way for developers to raise exceptions if the resolved symbol is `NULL`, and these exceptions can display meaningful messages and instructions to the users of extensions. -- https://bugs.ruby-lang.org/
Issue #20005 has been updated by kjtsanaktsidis (KJ Tsanaktsidis). Ah that makes sense - if you want to get potentially overridden symbols there isn’t a good way to do that at the moment. Plus I grant you the portability story around symbol exporting is messy, so wrapping that up would be handy. ---------------------------------------- Feature #20005: Add C API to return symbols of native extensions resolved from features https://bugs.ruby-lang.org/issues/20005#change-105337 * Author: tagomoris (Satoshi Tagomori) * Status: Open * Priority: Normal ---------------------------------------- I want an API to resolve symbols of other native extensions by a feature name and a symbol name (just like `dlsym`). (`rb_dln_resolve_symbol` is an example function name of this feature in the example below) ```c // "a_client_open" is a function defined in "/..../a_client.so", that should be loaded in Ruby beforehand. // in the extension to be "a.so" VALUE (*a_client_open)(VALUE); void Init_a(void) { a_client_open = (VALUE(*)(VALUE))rb_dln_resolve_symbol("a_client", "a_client_open"); // the return value may be NULL if the symbols is not found } ``` This API is to replace direct reference of external symbols. Currently, native extensions have to just call the function of other extensions directly on the assumption of: * The dependency native extension is already loaded * The symbol is defined correctly in the dependency (the version of dependency is correct&expected) Otherwise, it crashes. This API provides a way for developers to raise exceptions if the resolved symbol is `NULL`, and these exceptions can display meaningful messages and instructions to the users of extensions. -- https://bugs.ruby-lang.org/
Issue #20005 has been updated by tagomoris (Satoshi Tagomori). I noticed that `dln_sym` raises LoadError when the specified symbol is not found. But I expect this API to return NULL when the symbol is not found. Which one should be better? My expectation: libraries should raise LoadError under their own responsibility when this API returns NULL ---------------------------------------- Feature #20005: Add C API to return symbols of native extensions resolved from features https://bugs.ruby-lang.org/issues/20005#change-105410 * Author: tagomoris (Satoshi Tagomori) * Status: Open * Priority: Normal ---------------------------------------- I want an API to resolve symbols of other native extensions by a feature name and a symbol name (just like `dlsym`). (`rb_dln_resolve_symbol` is an example function name of this feature in the example below) ```c // "a_client_open" is a function defined in "/..../a_client.so", that should be loaded in Ruby beforehand. // in the extension to be "a.so" VALUE (*a_client_open)(VALUE); void Init_a(void) { a_client_open = (VALUE(*)(VALUE))rb_dln_resolve_symbol("a_client", "a_client_open"); // the return value may be NULL if the symbols is not found } ``` This API is to replace direct reference of external symbols. Currently, native extensions have to just call the function of other extensions directly on the assumption of: * The dependency native extension is already loaded * The symbol is defined correctly in the dependency (the version of dependency is correct&expected) Otherwise, it crashes. This API provides a way for developers to raise exceptions if the resolved symbol is `NULL`, and these exceptions can display meaningful messages and instructions to the users of extensions. -- https://bugs.ruby-lang.org/
Issue #20005 has been updated by nobu (Nobuyoshi Nakada). tagomoris (Satoshi Tagomori) wrote in #note-8:
I noticed that `dln_sym` raises LoadError when the specified symbol is not found. But I expect this API to return NULL when the symbol is not found. Which one should be better?
`dln_sym` has been used only to resolve and call immediately `Init_xxx` functions, and it would be reasonable to split error handling.
My expectation: libraries should raise LoadError under their own responsibility when this API returns NULL
What do you mean by "libraries" here? ```diff diff --git a/dln.c b/dln.c index bbed3af78ce..7c67fc2014f 100644 --- a/dln.c +++ b/dln.c @@ -419,32 +419,34 @@ dln_open(const char *file) static void * dln_sym(void *handle, const char *symbol) { - void *func; - const char *error; - #if defined(_WIN32) - char message[1024]; + return GetProcAddress(handle, symbol); +#elif defined(USE_DLN_DLOPEN) + return dlsym(handle, symbol); +#endif +} + +static void * +dln_sym_func(void *handle, const char *symbol) +{ + void *func = dln_sym(handle, symbol); - func = GetProcAddress(handle, symbol); if (func == NULL) { + const char *error; +#if defined(_WIN32) + char message[1024]; error = dln_strerror(); - goto failed; - } - #elif defined(USE_DLN_DLOPEN) - func = dlsym(handle, symbol); - if (func == NULL) { const size_t errlen = strlen(error = dln_strerror()) + 1; error = memcpy(ALLOCA_N(char, errlen), error, errlen); - goto failed; - } #endif - + dln_loaderror("%s - %s", error, symbol); + } return func; - - failed: - dln_loaderror("%s - %s", error, symbol); } + +#define dln_sym_call(rettype, argtype, handle, symbol) \ + (*(rettype (*)argtype)dln_sym_func(handle, symbol)) #endif #if defined(RUBY_DLN_CHECK_ABI) && defined(USE_DLN_DLOPEN) @@ -464,9 +466,8 @@ dln_load(const char *file) #ifdef RUBY_DLN_CHECK_ABI typedef unsigned long long abi_version_number; - typedef abi_version_number abi_version_func(void); - abi_version_func *abi_version_fct = (abi_version_func *)dln_sym(handle, EXTERNAL_PREFIX "ruby_abi_version"); - abi_version_number binary_abi_version = (*abi_version_fct)(); + abi_version_number binary_abi_version = + dln_sym_call(abi_version_number, (void), handle, EXTERNAL_PREFIX "ruby_abi_version")(); if (binary_abi_version != ruby_abi_version() && abi_check_enabled_p()) { dln_loaderror("incompatible ABI version of binary - %s", file); } @@ -474,10 +475,9 @@ dln_load(const char *file) char *init_fct_name; init_funcname(&init_fct_name, file); - void (*init_fct)(void) = (void(*)(void))dln_sym(handle, init_fct_name); /* Call the init code */ - (*init_fct)(); + dln_sym_call(void, (void), handle, init_fct_name)(); return handle; ``` ---------------------------------------- Feature #20005: Add C API to return symbols of native extensions resolved from features https://bugs.ruby-lang.org/issues/20005#change-105412 * Author: tagomoris (Satoshi Tagomori) * Status: Open * Priority: Normal ---------------------------------------- I want an API to resolve symbols of other native extensions by a feature name and a symbol name (just like `dlsym`). (`rb_dln_resolve_symbol` is an example function name of this feature in the example below) ```c // "a_client_open" is a function defined in "/..../a_client.so", that should be loaded in Ruby beforehand. // in the extension to be "a.so" VALUE (*a_client_open)(VALUE); void Init_a(void) { a_client_open = (VALUE(*)(VALUE))rb_dln_resolve_symbol("a_client", "a_client_open"); // the return value may be NULL if the symbols is not found } ``` This API is to replace direct reference of external symbols. Currently, native extensions have to just call the function of other extensions directly on the assumption of: * The dependency native extension is already loaded * The symbol is defined correctly in the dependency (the version of dependency is correct&expected) Otherwise, it crashes. This API provides a way for developers to raise exceptions if the resolved symbol is `NULL`, and these exceptions can display meaningful messages and instructions to the users of extensions. -- https://bugs.ruby-lang.org/
Issue #20005 has been updated by tagomoris (Satoshi Tagomori). nobu (Nobuyoshi Nakada) wrote in #note-9:
`dln_sym` has been used only to resolve and call immediately `Init_xxx` functions, and it would be reasonable to split error handling.
That makes sense.
My expectation: libraries should raise LoadError under their own responsibility when this API returns NULL
What do you mean by "libraries" here?
I meant extensions, that call this API. The diff you pasted is really helpful. Just one point I have a question: ```diff +#define dln_sym_call(rettype, argtype, handle, symbol) \ + (*(rettype (*)argtype)dln_sym_func(handle, symbol)) ``` This actually does not call the symbol but returns a callable symbol. So I think it's better to either: A. rename it to `dln_sym_callable` B. add `()` at the end of the line, and remove `()` from the code about `"ruby_abi_version"` and `init_fct_name`. @nobu What do you think? ---------------------------------------- Feature #20005: Add C API to return symbols of native extensions resolved from features https://bugs.ruby-lang.org/issues/20005#change-105430 * Author: tagomoris (Satoshi Tagomori) * Status: Open * Priority: Normal ---------------------------------------- I want an API to resolve symbols of other native extensions by a feature name and a symbol name (just like `dlsym`). (`rb_dln_resolve_symbol` is an example function name of this feature in the example below) ```c // "a_client_open" is a function defined in "/..../a_client.so", that should be loaded in Ruby beforehand. // in the extension to be "a.so" VALUE (*a_client_open)(VALUE); void Init_a(void) { a_client_open = (VALUE(*)(VALUE))rb_dln_resolve_symbol("a_client", "a_client_open"); // the return value may be NULL if the symbols is not found } ``` This API is to replace direct reference of external symbols. Currently, native extensions have to just call the function of other extensions directly on the assumption of: * The dependency native extension is already loaded * The symbol is defined correctly in the dependency (the version of dependency is correct&expected) Otherwise, it crashes. This API provides a way for developers to raise exceptions if the resolved symbol is `NULL`, and these exceptions can display meaningful messages and instructions to the users of extensions. -- https://bugs.ruby-lang.org/
Issue #20005 has been updated by nobu (Nobuyoshi Nakada). tagomoris (Satoshi Tagomori) wrote in #note-10:
This actually does not call the symbol but returns a callable symbol. So I think it's better to either: A. rename it to `dln_sym_callable` B. add `()` at the end of the line, and remove `()` from the code about `"ruby_abi_version"` and `init_fct_name`.
For B, that macro also needs an argument like `argtype`: ```C dln_sym_call(void, (void), handle, init_fct_name, ()); ``` I'm for A a little, right now. ---------------------------------------- Feature #20005: Add C API to return symbols of native extensions resolved from features https://bugs.ruby-lang.org/issues/20005#change-105521 * Author: tagomoris (Satoshi Tagomori) * Status: Open * Priority: Normal ---------------------------------------- I want an API to resolve symbols of other native extensions by a feature name and a symbol name (just like `dlsym`). (`rb_dln_resolve_symbol` is an example function name of this feature in the example below) ```c // "a_client_open" is a function defined in "/..../a_client.so", that should be loaded in Ruby beforehand. // in the extension to be "a.so" VALUE (*a_client_open)(VALUE); void Init_a(void) { a_client_open = (VALUE(*)(VALUE))rb_dln_resolve_symbol("a_client", "a_client_open"); // the return value may be NULL if the symbols is not found } ``` This API is to replace direct reference of external symbols. Currently, native extensions have to just call the function of other extensions directly on the assumption of: * The dependency native extension is already loaded * The symbol is defined correctly in the dependency (the version of dependency is correct&expected) Otherwise, it crashes. This API provides a way for developers to raise exceptions if the resolved symbol is `NULL`, and these exceptions can display meaningful messages and instructions to the users of extensions. -- https://bugs.ruby-lang.org/
Issue #20005 has been updated by tagomoris (Satoshi Tagomori). nobu (Nobuyoshi Nakada) wrote in #note-11:
I'm for A a little, right now.
I see. I'll change the name later. ---------------------------------------- Feature #20005: Add C API to return symbols of native extensions resolved from features https://bugs.ruby-lang.org/issues/20005#change-105522 * Author: tagomoris (Satoshi Tagomori) * Status: Open * Priority: Normal ---------------------------------------- I want an API to resolve symbols of other native extensions by a feature name and a symbol name (just like `dlsym`). (`rb_dln_resolve_symbol` is an example function name of this feature in the example below) ```c // "a_client_open" is a function defined in "/..../a_client.so", that should be loaded in Ruby beforehand. // in the extension to be "a.so" VALUE (*a_client_open)(VALUE); void Init_a(void) { a_client_open = (VALUE(*)(VALUE))rb_dln_resolve_symbol("a_client", "a_client_open"); // the return value may be NULL if the symbols is not found } ``` This API is to replace direct reference of external symbols. Currently, native extensions have to just call the function of other extensions directly on the assumption of: * The dependency native extension is already loaded * The symbol is defined correctly in the dependency (the version of dependency is correct&expected) Otherwise, it crashes. This API provides a way for developers to raise exceptions if the resolved symbol is `NULL`, and these exceptions can display meaningful messages and instructions to the users of extensions. -- https://bugs.ruby-lang.org/
Issue #20005 has been updated by matz (Yukihiro Matsumoto). Accepted (with a new name). Matz. ---------------------------------------- Feature #20005: Add C API to return symbols of native extensions resolved from features https://bugs.ruby-lang.org/issues/20005#change-105556 * Author: tagomoris (Satoshi Tagomori) * Status: Open * Priority: Normal ---------------------------------------- I want an API to resolve symbols of other native extensions by a feature name and a symbol name (just like `dlsym`). (`rb_dln_resolve_symbol` is an example function name of this feature in the example below) ```c // "a_client_open" is a function defined in "/..../a_client.so", that should be loaded in Ruby beforehand. // in the extension to be "a.so" VALUE (*a_client_open)(VALUE); void Init_a(void) { a_client_open = (VALUE(*)(VALUE))rb_dln_resolve_symbol("a_client", "a_client_open"); // the return value may be NULL if the symbols is not found } ``` This API is to replace direct reference of external symbols. Currently, native extensions have to just call the function of other extensions directly on the assumption of: * The dependency native extension is already loaded * The symbol is defined correctly in the dependency (the version of dependency is correct&expected) Otherwise, it crashes. This API provides a way for developers to raise exceptions if the resolved symbol is `NULL`, and these exceptions can display meaningful messages and instructions to the users of extensions. -- https://bugs.ruby-lang.org/
participants (4)
-
kjtsanaktsidis (KJ Tsanaktsidis) -
matz (Yukihiro Matsumoto)
-
nobu (Nobuyoshi Nakada)
-
tagomoris (Satoshi Tagomori)