Rufus scheduler and database connection issues.

[dmitry-sinina]

After database restart rufus scheduler failing jobs:

Feb 10 09:03:00 hostname ruby[522]: { 836547360 rufus-scheduler intercepted an error:
Feb 10 09:03:00 hostname ruby[522]:   836547360   job:
Feb 10 09:03:00 hostname ruby[522]:   836547360     Rufus::Scheduler::CronJob "* * * * *" {:overlap=>false, :name=>"AccountBalanceNotify", :timeout=>1200}
Feb 10 09:03:00 hostname ruby[522]:   836547360   error:
Feb 10 09:03:00 hostname ruby[522]:   836547360     836547360
Feb 10 09:03:00 hostname ruby[522]:   836547360     ActiveRecord::StatementInvalid
Feb 10 09:03:00 hostname ruby[522]:   836547360     PG::UnableToSend: no connection to the server
Feb 10 09:03:00 hostname ruby[522]:   836547360       (eval):39:in `exec_prepared'
Feb 10 09:03:00 hostname ruby[522]:   836547360       (eval):39:in `exec_prepared'
Feb 10 09:03:00 hostname ruby[522]:   836547360       /opt/yeti-web/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.4/lib/active_record/connection_adapters/postgresql_adapter.rb:687:in `block (2 levels) in exec_cache'
Feb 10 09:03:00 hostname ruby[522]:   836547360       /opt/yeti-web/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.4/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
Feb 10 09:03:00 hostname ruby[522]:   836547360       /opt/yeti-web/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.4/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
Feb 10 09:03:00 hostname ruby[522]:   836547360       /opt/yeti-web/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.4/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
Feb 10 09:03:00 hostname ruby[522]:   836547360       /opt/yeti-web/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.4/lib/active_record/connection_adapters/postgresql_adapter.rb:686:in `block in exec_cache'
Feb 10 09:03:00 hostname ruby[522]:   836547360       /opt/yeti-web/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.4/lib/active_record/connection_adapters/abstract_adapter.rb:696:in `block (2 levels) in log'
Feb 10 09:03:00 hostname ruby[522]:   836547360       /opt/yeti-web/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
Feb 10 09:03:00 hostname ruby[522]:   836547360       /opt/yeti-web/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
Feb 10 09:03:00 hostname ruby[522]:   836547360       /opt/yeti-web/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'

Looks like rufus doesn't try to restore database connections and jobs just failing every time.

possible solutions:

• check database connections before job execution and reconnect in case of problem
• crash rufus-scheduler completely(systemd will restart it)

[senid231]

we always establish connections before job run
lib/yeti_scheduler.rb:9-10

but maybe after the connection is broken due to network issues, it does not release properly and is being reused.
as solution, you can try to release the connection before establishing it (before line 9)

ApplicationRecord.connection_pool.release_connection
Cdr::Base.connection_pool.release_connection
``