[ruby-core:113455] [Ruby master Bug#19638] Multithread is not working as expected after gem fix
Issue #19638 has been reported by leonard100 (Leo Camp). ---------------------------------------- Bug #19638: Multithread is not working as expected after gem fix https://bugs.ruby-lang.org/issues/19638 * Author: leonard100 (Leo Camp) * Status: Open * Priority: Normal * ruby -v: 2.7.6 * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Hi, Im currently working on a ruby application that uses Ruby 2.7, ActiveRecord 7 and IBM_DB gem 5.4.0(with the fix https://bugs.ruby-lang.org/issues/19527) The application requires using ActiveRecord::Base for the connection_pool, Threads and the IBM_DB gem as an adapter to connect to the database but using ActiveRecord. The application requires running multiple queries at the same time, limited by the pool size. Previously using activerecord-jdbc-adapter and Jruby 1.7 which could run multiple connections and for each connection run a query using the pool and threads, that means 5 queries = 5 connections = 5 threads running them Now using the IBM_DB gem, you can execute the connections and queries with different ones completing the process, the problem is that it does it one by one instead of several at the same time, being the same code as before, it hangs just at the moment the one that executes the query in the thread, waits until it ends to go to the next one, this means 5 queries = 1 connection (different in each query) = 1 thread executing it (different in each query) We used the fixed version with the GC issue mentioned in this post https://bugs.ruby-lang.org/issues/19527 where we changed and recompiled the gem as follows: ``` ruby Before /IBM_DB_Adapter/ibm_db/ext/ibm_db.c _ruby_ibm_db_mark_stmt_struct(stmt_handle *handle) static inline VALUE ibm_Ruby_Thread_Call(rb_blocking_function_t *func, void *data1, rb_unblock_function_t *ubf, void *data2) { void *(*f)(void*) = (void *(*)(void*))func; return (VALUE)rb_thread_call_without_gvl(f, data1, ubf, data2); } After static inline VALUE ibm_Ruby_Thread_Call(rb_blocking_function_t *func, void *data1, rb_unblock_function_t *ubf, void *data2) { + return func(data1); } ``` This solved the GC problem, but since it is a method that handles the Threads when it is executed, it makes me believe that this Fix that was made to the method may be causing that when a query is executed it hangs there until that query ends and proceed with another thread/connection/query instead of executing it and leaving it running in the background while it goes to do another thread/connection/query I made this test code where I do the same as the application on a smaller scale, but the exact same case mentioned above happens, my doubt is the problem will be in this code or in the previous fix: ``` ruby require 'ibm_db' require 'active_record' $control_tables_schema = "" $db_config = { adapter: 'ibm_db', schema: $control_tables_schema, database: '', username: '', password: '', checkout_timeout: 20, pool: 20 # number of simultaneous connections } conn = nil condsql = nil condsql = ["SQL COMMAND", "SQL COMMAND", "SQL COMMAND"] condsql.each do |auxsql| threads = Thread.new(auxsql) do |command| conn = ActiveRecord::Base.establish_connection($db_config) res = nil conn.with_connection do |connection_cond| res = connection_cond.execute(auxsql) end #puts "************************************" puts conn.stat puts conn.connections() puts owner_thread = Thread.current puts res puts "************************************" #Output looks like this, no errors, just one by one instead of all at the same time #{:size=>20, :connections=>1, :busy=>0, :dead=>0, :idle=>1, :waiting=>0, :checkout_timeout=>20.0} #<ActiveRecord::ConnectionAdapters::IBM_DBAdapter:0x00007faf2852d150> #<Thread:0x00007faf44248b80 test.rb:26 run> end threads.join if conn then conn.release_connection(owner_thread = Thread.current) conn.clear_active_connections! rescue nil conn.clear_stale_cached_connections! rescue nil end end ``` -- https://bugs.ruby-lang.org/
Issue #19638 has been updated by jeremyevans0 (Jeremy Evans). Status changed from Open to Third Party's Issue This is likely another bug in the ibm_db gem. Please report the bug to the ibm_db developers: https://github.com/ibmdb/ruby-ibmdb/issues/new If you can reproduce the issue without using the ibm_db gem, on a supported version of Ruby (3.1 or 3.2), please reply with a minimal self-contained reproducible example. ---------------------------------------- Bug #19638: Multithread is not working as expected after gem fix https://bugs.ruby-lang.org/issues/19638#change-103034 * Author: leonard100 (Leo Camp) * Status: Third Party's Issue * Priority: Normal * ruby -v: 2.7.6 * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Hi, Im currently working on a ruby application that uses Ruby 2.7, ActiveRecord 7 and IBM_DB gem 5.4.0(with the fix https://bugs.ruby-lang.org/issues/19527) The application requires using ActiveRecord::Base for the connection_pool, Threads and the IBM_DB gem as an adapter to connect to the database but using ActiveRecord. The application requires running multiple queries at the same time, limited by the pool size. Previously using activerecord-jdbc-adapter and Jruby 1.7 which could run multiple connections and for each connection run a query using the pool and threads, that means 5 queries = 5 connections = 5 threads running them Now using the IBM_DB gem, you can execute the connections and queries with different ones completing the process, the problem is that it does it one by one instead of several at the same time, being the same code as before, it hangs just at the moment the one that executes the query in the thread, waits until it ends to go to the next one, this means 5 queries = 1 connection (different in each query) = 1 thread executing it (different in each query) We used the fixed version with the GC issue mentioned in this post https://bugs.ruby-lang.org/issues/19527 where we changed and recompiled the gem as follows: ``` ruby Before /IBM_DB_Adapter/ibm_db/ext/ibm_db.c _ruby_ibm_db_mark_stmt_struct(stmt_handle *handle) static inline VALUE ibm_Ruby_Thread_Call(rb_blocking_function_t *func, void *data1, rb_unblock_function_t *ubf, void *data2) { void *(*f)(void*) = (void *(*)(void*))func; return (VALUE)rb_thread_call_without_gvl(f, data1, ubf, data2); } After static inline VALUE ibm_Ruby_Thread_Call(rb_blocking_function_t *func, void *data1, rb_unblock_function_t *ubf, void *data2) { + return func(data1); } ``` This solved the GC problem, but since it is a method that handles the Threads when it is executed, it makes me believe that this Fix that was made to the method may be causing that when a query is executed it hangs there until that query ends and proceed with another thread/connection/query instead of executing it and leaving it running in the background while it goes to do another thread/connection/query I made this test code where I do the same as the application on a smaller scale, but the exact same case mentioned above happens, my doubt is the problem will be in this code or in the previous fix: ``` ruby require 'ibm_db' require 'active_record' $control_tables_schema = "" $db_config = { adapter: 'ibm_db', schema: $control_tables_schema, database: '', username: '', password: '', checkout_timeout: 20, pool: 20 # number of simultaneous connections } conn = nil condsql = nil condsql = ["SQL COMMAND", "SQL COMMAND", "SQL COMMAND"] condsql.each do |auxsql| threads = Thread.new(auxsql) do |command| conn = ActiveRecord::Base.establish_connection($db_config) res = nil conn.with_connection do |connection_cond| res = connection_cond.execute(auxsql) end #puts "************************************" puts conn.stat puts conn.connections() puts owner_thread = Thread.current puts res puts "************************************" #Output looks like this, no errors, just one by one instead of all at the same time #{:size=>20, :connections=>1, :busy=>0, :dead=>0, :idle=>1, :waiting=>0, :checkout_timeout=>20.0} #<ActiveRecord::ConnectionAdapters::IBM_DBAdapter:0x00007faf2852d150> #<Thread:0x00007faf44248b80 test.rb:26 run> end threads.join if conn then conn.release_connection(owner_thread = Thread.current) conn.clear_active_connections! rescue nil conn.clear_stale_cached_connections! rescue nil end end ``` -- https://bugs.ruby-lang.org/
participants (2)
-
jeremyevans0 (Jeremy Evans) -
leonard100 (Leo Camp)