feat: Solid Queue Integration (#199)

This adds an adapter to integrate with SolidQueue, reporting job queue time and busy metrics (if enabled) to Judoscale for autoscaling.

SolidQueue is currently on v0.3, still pretty early on, and there's still some things being figured out, but there's early adoption and we expect more as it becomes a Rails recommendation / default in the future. It only works with Rails v7.1+ and Ruby 2.7+, so that's what this adapter will support initially.

We'll be collecting queue time / latency via the "ready executions" table, and busy via the "claimed executions" table.

SolidQueue moves jobs between different tables as they change "status", in other words, while all jobs have a representation on the main "jobs" table, they also get a record on an associated table that may represent what's happening to them: when they're ready to be picked up for work, they go to "ready executions", when they're claimed by a process worker to be performed, they go to "claimed executions", and if there's a failure (that's not retired by Active Job), they go to "failed executions"; if they're scheduled to run in the future, they go to "scheduled executions" (or if they're being retried by AJ, which is essentially re-scheduling them in the future, until it succeeds or gives up retrying and blows up back to SolidQueue.)

When jobs are finished successfully, they are flagged with a "finished_at" column on the main "jobs" table. As the jobs move from one to the other "execution" status in the workflow, their previous record is destroyed, so there should be really only one of those "execution" representations at one point in time. (i.e. a job is either scheduled, ready, claimed, failed)

There's also the concept of recurring executions, which are created via config (a cron-like setup), and eventually get added to "ready executions" for every recur.

And finally, there's blocked executions. Jobs can be configured with a concurrency limit, i.e. run only one job at a time, or one job with this set of arguments, or up to X jobs concurrently, etc., and if more jobs are enqueued, instead of going to "ready", they go to "blocked". When jobs are finished, they check for blocked jobs to unblock them, and there's also an additional dispatcher that checks for blocked jobs on a schedule. While initially I thought it'd make sense to consider these for the latency calculation, the more I thought about and played with it, the more it came to mind that having a big list of blocked jobs doesn't mean a need to autoscale: you might simply be limiting the concurrency of those jobs to a point where many are getting enqueued at certain points, but just a few get processed due to the limits imposed. This could cause the blocked execution table to grow temporarily, causing those blocked jobs to have "increased latency", but autoscaling up might be wrong in this case, since more processing power won't make those jobs complete any faster -- they're still limited by their concurrency setup.

---------

Co-authored-by: Adam McCrea <adam@adamlogic.com>

.github/workflows/judoscale-solid_queue.yml

@@ -0,0 +1,45 @@
+name: judoscale-solid_queue tests
+defaults:
+  run:
+    working-directory: judoscale-solid_queue
+on:
+  push:
+    branches:
+      - main
+  pull_request:
+jobs:
+  test:
+    strategy:
+      fail-fast: false
+      matrix:
+        gemfile:
+          - Gemfile
+        ruby:
+          - "2.7"
+          - "3.0"
+          - "3.1"
+          - "3.2"
+          - "3.3"
+        # exclude:
+        #   -
+    runs-on: ubuntu-latest
+    env: # $BUNDLE_GEMFILE must be set at the job level, so it is set for all steps
+      BUNDLE_GEMFILE: ${{ github.workspace }}/judoscale-solid_queue/${{ matrix.gemfile }}
+    services:
+      db:
+        image: postgres:latest
+        env:
+          POSTGRES_HOST_AUTH_METHOD: trust
+        ports: ["5432:5432"]
+        options: >-
+          --health-cmd pg_isready
+          --health-interval 10s
+          --health-timeout 5s
+          --health-retries 5
+    steps:
+      - uses: actions/checkout@v4
+      - uses: ruby/setup-ruby@v1
+        with:
+          ruby-version: ${{ matrix.ruby }}
+          bundler-cache: true # runs bundle install and caches installed gems automatically
+      - run: bundle exec rake

README.md

@@ -6,6 +6,7 @@
 [![Build Status: judoscale-good_job](https://github.com/judoscale/judoscale-ruby/actions/workflows/judoscale-good_job-test.yml/badge.svg)](https://github.com/judoscale/judoscale-ruby/actions)
 [![Build Status: judoscale-que](https://github.com/judoscale/judoscale-ruby/actions/workflows/judoscale-que-test.yml/badge.svg)](https://github.com/judoscale/judoscale-ruby/actions)
 [![Build Status: judoscale-sidekiq](https://github.com/judoscale/judoscale-ruby/actions/workflows/judoscale-sidekiq-test.yml/badge.svg)](https://github.com/judoscale/judoscale-ruby/actions)
+[![Build Status: judoscale-solid_queue](https://github.com/judoscale/judoscale-ruby/actions/workflows/judoscale-solid_queue-test.yml/badge.svg)](https://github.com/judoscale/judoscale-ruby/actions)
 
 These gems works together with the [Judoscale](https://judoscale.com) Heroku add-on to scale your web and worker dynos automatically. They gather a minimal set of metrics for each request and job queue, and periodically posts this data asynchronously to the Judoscale API.
 
@@ -27,6 +28,7 @@ gem "judoscale-rails"
 # gem "judoscale-delayed_job"
 # gem "judoscale-good_job"
 # gem "judoscale-que"
+# gem "judoscale-solid_queue"
 ```
 
 _If you're using a background job queue, make sure you include the corresponding judoscale-\* gem as well._
@@ -52,14 +54,15 @@ The middleware will start the async reporter when it processes the first request
 
 ## Worker adapters
 
-Judoscale will autoscale your worker dynos! Four job backends are supported: Sidekiq, Delayed Job, Good Job, and Que. Be sure to install the gem specific to your job backend:
+Judoscale will autoscale your worker dynos! The following job backends are supported: Sidekiq, Resque, Delayed Job, Good Job, Que, and Solid Queue. Be sure to install the gem specific to your job backend:
 
 ```ruby
 gem "judoscale-sidekiq"
 gem "judoscale-resque"
 gem "judoscale-delayed_job"
 gem "judoscale-good_job"
 gem "judoscale-que"
+gem "judoscale-solid_queue"
 ```
 
 For most apps, no additional configuration is needed. See the [configuration](#configuration) section below for all available options.

judoscale-good_job/test/test_helper.rb

@@ -10,6 +10,7 @@
 require "action_controller"
 
 class TestRailsApp < Rails::Application
+  config.load_defaults "#{Rails::VERSION::MAJOR}.#{Rails::VERSION::MINOR}"
   config.secret_key_base = "test-secret"
   config.eager_load = false
   config.logger = ::Logger.new(StringIO.new, progname: "rails-app")

judoscale-rails/test/test_helper.rb

@@ -11,6 +11,7 @@
 require "action_controller"
 
 class TestRailsApp < Rails::Application
+  config.load_defaults "#{Rails::VERSION::MAJOR}.#{Rails::VERSION::MINOR}"
   config.secret_key_base = "test-secret"
   config.eager_load = false
   config.logger = ::Logger.new(StringIO.new, progname: "rails-app")

judoscale-solid_queue/Gemfile

@@ -0,0 +1,9 @@
+source "https://rubygems.org"
+
+gemspec name: "judoscale-solid_queue"
+
+gem "judoscale-ruby", path: "../judoscale-ruby"
+gem "activerecord"
+gem "pg"
+gem "minitest"
+gem "rake"

judoscale-solid_queue/Rakefile

@@ -0,0 +1,11 @@
+# frozen_string_literal: true
+
+require "rake/testtask"
+
+Rake::TestTask.new(:test) do |t|
+  t.libs << "lib"
+  t.libs << "test"
+  t.test_files = FileList["test/**/*_test.rb"]
+end
+
+task default: :test

judoscale-solid_queue/judoscale-solid_queue.gemspec

@@ -0,0 +1,30 @@
+lib = File.expand_path("../lib", __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require "judoscale/solid_queue/version"
+
+Gem::Specification.new do |spec|
+  spec.name = "judoscale-solid_queue"
+  spec.version = Judoscale::SolidQueue::VERSION
+  spec.authors = ["Adam McCrea", "Carlos Antonio da Silva", "Jon Sullivan"]
+  spec.email = ["hello@judoscale.com"]
+
+  spec.summary = "This gem provides SolidQueue integration with the Judoscale autoscaling add-on for Heroku."
+  spec.homepage = "https://judoscale.com"
+  spec.license = "MIT"
+
+  spec.metadata = {
+    "homepage_uri" => "https://judoscale.com",
+    "bug_tracker_uri" => "https://github.com/judoscale/judoscale-ruby/issues",
+    "documentation_uri" => "https://judoscale.com/docs",
+    "changelog_uri" => "https://github.com/judoscale/judoscale-ruby/blob/main/CHANGELOG.md",
+    "source_code_uri" => "https://github.com/judoscale/judoscale-ruby"
+  }
+
+  spec.files = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
+  spec.require_paths = ["lib"]
+
+  spec.required_ruby_version = ">= 2.7.0"
+
+  spec.add_dependency "judoscale-ruby", Judoscale::SolidQueue::VERSION
+  spec.add_dependency "solid_queue", ">= 0.3"
+end

judoscale-solid_queue/lib/judoscale-solid_queue.rb

@@ -0,0 +1,3 @@
+# frozen_string_literal: true
+
+require "judoscale/solid_queue"

judoscale-solid_queue/lib/judoscale/solid_queue.rb

@@ -0,0 +1,15 @@
+# frozen_string_literal: true
+
+require "solid_queue"
+require "judoscale-ruby"
+require "judoscale/config"
+require "judoscale/solid_queue/version"
+require "judoscale/solid_queue/metrics_collector"
+
+Judoscale.add_adapter :"judoscale-solid_queue",
+  {
+    adapter_version: Judoscale::SolidQueue::VERSION,
+    framework_version: ::SolidQueue::VERSION
+  },
+  metrics_collector: Judoscale::SolidQueue::MetricsCollector,
+  expose_config: Judoscale::Config::JobAdapterConfig.new(:solid_queue)

judoscale-solid_queue/lib/judoscale/solid_queue/metrics_collector.rb

@@ -0,0 +1,63 @@
+# frozen_string_literal: true
+
+require "judoscale/job_metrics_collector"
+require "judoscale/job_metrics_collector/active_record_helper"
+require "judoscale/metric"
+
+module Judoscale
+  module SolidQueue
+    class MetricsCollector < Judoscale::JobMetricsCollector
+      include ActiveRecordHelper
+
+      def self.adapter_config
+        Judoscale::Config.instance.solid_queue
+      end
+
+      def self.collect?(config)
+        super && ActiveRecordHelper.table_exists_for_model?(::SolidQueue::Job)
+      end
+
+      def initialize
+        super
+
+        queue_names = run_silently do
+          ::SolidQueue::Job.distinct.pluck(:queue_name)
+        end
+        self.queues |= queue_names
+      end
+
+      def collect
+        metrics = []
+        time = Time.now.utc
+
+        oldest_execution_time_by_queue = run_silently do
+          ::SolidQueue::ReadyExecution.group(:queue_name).minimum(:created_at)
+        end
+        self.queues |= oldest_execution_time_by_queue.keys
+
+        if track_busy_jobs?
+          busy_count_by_queue = run_silently do
+            ::SolidQueue::Job.joins(:claimed_execution).group(:queue_name).count
+          end
+          self.queues |= busy_count_by_queue.keys
+        end
+
+        queues.each do |queue|
+          run_at = oldest_execution_time_by_queue[queue]
+          latency_ms = run_at ? ((time - run_at) * 1000).ceil : 0
+          latency_ms = 0 if latency_ms < 0
+
+          metrics.push Metric.new(:qt, latency_ms, time, queue)
+
+          if track_busy_jobs?
+            busy_count = busy_count_by_queue[queue] || 0
+            metrics.push Metric.new(:busy, busy_count, Time.now, queue)
+          end
+        end
+
+        log_collection(metrics)
+        metrics
+      end
+    end
+  end
+end

judoscale-solid_queue/lib/judoscale/solid_queue/version.rb

@@ -0,0 +1,7 @@
+# frozen_string_literal: true
+
+module Judoscale
+  module SolidQueue
+    VERSION = "1.5.4"
+  end
+end

judoscale-solid_queue/lib/rails-autoscale-solid_queue.rb

@@ -0,0 +1,3 @@
+# frozen_string_literal: true
+
+require "judoscale/solid_queue"

judoscale-solid_queue/rails-autoscale-solid_queue.gemspec

@@ -0,0 +1,30 @@
+lib = File.expand_path("../lib", __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require "judoscale/solid_queue/version"
+
+Gem::Specification.new do |spec|
+  spec.name = "rails-autoscale-solid_queue"
+  spec.version = Judoscale::SolidQueue::VERSION
+  spec.authors = ["Adam McCrea", "Carlos Antonio da Silva", "Jon Sullivan"]
+  spec.email = ["hello@judoscale.com"]
+
+  spec.summary = "This gem provides SolidQueue integration with the Judoscale autoscaling add-on for Heroku."
+  spec.homepage = "https://judoscale.com"
+  spec.license = "MIT"
+
+  spec.metadata = {
+    "homepage_uri" => "https://judoscale.com",
+    "bug_tracker_uri" => "https://github.com/judoscale/judoscale-ruby/issues",
+    "documentation_uri" => "https://judoscale.com/docs",
+    "changelog_uri" => "https://github.com/judoscale/judoscale-ruby/blob/main/CHANGELOG.md",
+    "source_code_uri" => "https://github.com/judoscale/judoscale-ruby"
+  }
+
+  spec.files = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
+  spec.require_paths = ["lib"]
+
+  spec.required_ruby_version = ">= 2.7.0"
+
+  spec.add_dependency "rails-autoscale-core", Judoscale::SolidQueue::VERSION
+  spec.add_dependency "solid_queue", ">= 0.3"
+end

judoscale-solid_queue/test/adapter_test.rb

@@ -0,0 +1,36 @@
+# frozen_string_literal: true
+
+require "test_helper"
+require "judoscale/report"
+
+module Judoscale
+  describe SolidQueue do
+    it "adds itself as an adapter with information to be reported to the Judoscale API" do
+      adapter = Judoscale.adapters.detect { |adapter| adapter.identifier == :"judoscale-solid_queue" }
+      _(adapter).wont_be_nil
+      _(adapter.metrics_collector).must_equal Judoscale::SolidQueue::MetricsCollector
+
+      report = ::Judoscale::Report.new(Judoscale.adapters, Judoscale::Config.instance, [])
+      _(report.as_json[:adapters]).must_include(:"judoscale-solid_queue")
+    end
+
+    it "sets up a config property for the library" do
+      config = Config.instance
+      _(config.solid_queue.enabled).must_equal true
+      _(config.solid_queue.max_queues).must_equal 20
+      _(config.solid_queue.queues).must_equal []
+      _(config.solid_queue.track_busy_jobs).must_equal false
+
+      Judoscale.configure do |config|
+        config.solid_queue.queues = %w[test drive]
+        config.solid_queue.track_busy_jobs = true
+      end
+
+      _(config.solid_queue.queues).must_equal %w[test drive]
+      _(config.solid_queue.track_busy_jobs).must_equal true
+
+      report = ::Judoscale::Report.new(Judoscale.adapters, Judoscale::Config.instance, [])
+      _(report.as_json[:config]).must_include(:solid_queue)
+    end
+  end
+end